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ABSTRACT 

This  thesis  evaluates  the  Navy's  test  and  evaluation  (T&E)  process 
for  air-launched  guided  missiles,  identifies  T&E  management  problems, 
and  proposes  utilization  of  a  particular  test  program  strategy.   Many 
changes  have  been  made  to  improve  the  T&E  process,  but  some  of  these 
have  resulted  in  costly  inefficiencies.   Contributing  problems  include: 
the  operational  test  agencies  do  not  participate  adequately  in  early 
test  planning,  excess  duplication  exists  in  testing  done  by  major  par- 
ticipants in  the  Navy  air-launched  guided  missile  T&E  process,  lack  of 
definition  of  a  mission  profile  leads  to  improper  and  inadequate  test- 
ing of  air-launched  guided  missiles,  and  numerous  other  T&E  management 
problems. 

The  utilization  of  an  Integrated  Test  Program  approach  for  air- 
launched  guided  missile  T&E  is  proposed.   Test  data  from  contractor 
demonstration  tests,  limited  TECHEVAL,  and  OPEVAL  would  be  pooled  to 
confirm  compliance   with  specification  requirements   and  verify  opera- 
tional effectiveness  and  suitability.   Test  assets  and  other  resources 
would  be  conserved  and  overall  T&E  schedule  improved. 
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I.   INTRODUCTION 

A.   BACKGROUND 

The  acquisition  process  for  defense  weapon  systems  came  under  in- 
tense criticism  in  the  late  1960s  and  early  1970s  because  of  costly 
development  overruns,  schedule  slippage,  and  system  performance  short- 
comings.  The  President's  Blue  Ribbon  Defense  Panel,  the  General 
Accounting  Office,  the  Congress's  Commission  on  Government  Procurement, 
and  the  Armed  Services  Committees  of  the  House  of  Representatives  and 
Senate  have  consistently  identified  test  and  evaluation  as  a  major 
problem  area.   The  Department  of  Defense  (DoD)  responded  to  the  criti- 
cism by  issuing  new  guidelines  and  directives  instituting  procedures 
which  put  a  new  perspective  on  the  acquisition  process.   A  major  out- 
come of  the  new  procedures  was  the  increased  emphasis  of  test  and 
evaluation  (TOE) .   Of  concern  was  the  fact  that  T&E  had  not  commanded 
the  importance,  stature,  or  priority  that  it  must  have  if  it  is  to  be 
a  primary  source  of  information  on  the  progress  of  major  defense  sys- 
tems and  for  decision-making. 

As  a  result,  the  new  directives  stated  that  T&E  will  commence  as 
early  as  possible  and  be  conducted  throughout  the  system  acquisition 
process,  and  that  program  schedules  and  milestone  decisions  will  be 
based  upon  accomplishment  and  assessment  of  the  program's  T&E  efforts. 
Also  implemented  was  the  establishment  of  a  T&E  organization  in  the 
Offices  of  the  Secretary  of  Defense  (OSD)  to  oversee  defense  T&E  activ- 
ities.  It  was  further  required  that  a  primary  role  would  be  played  by 
independent  test  agencies  within  DoD  components  in  the  accomplishment 
of  T&E  [Refs.  1  and  2]  . 
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Implementation  of  the  guidelines  and  directives  has  encountered 
some  difficulties.   T&E  is  costly,  difficult,  and  time  consuming. 
There  are  many  people  involved  and  genuine  competition  exists  within 
DoD  between  agencies,  between  DoD  agencies  and  contractors,  and  be- 
tween contractors  for  scarce  defense  dollars. 

However,  the  value  of  T&E  cannot  be  overlooked.   In  a  recent  over- 
view statement  to  Congress,  the  Deputy  Director  (Test  and  Evaluation) 
of  the  Office  of  Defense  Research  and  Engineering  said  that  T&E  was  a 
good  investment  from  which  essential  information  was  obtained  to 
effectively  manage  the  research,  development,  and  procurement  of 
defense  systems.   By  insuring  that  only  proven,  fully-effective  weapon 
systems  are  deployed,  T&E  allows  us  to  obtain  maximum  military  capa- 
bility for  our  defense  dollars  [Ref .  3] . 

3.       PURPOSE   A1ID   OBJECTIVES 

The  purpose  of  this  thesis  is  to  examine  the  management  policy  of 
T&E  during  this  new  era  of  T&E  emphasis  and  some  of  its  associated 
problem  areas,  and  then  to  relate  the  policy  and  problems  to  an  Air 
Launched  Guided  Missile  (ALGM)  program. 

The  primary  objective  of  the  thesis  is  to  provide  guidance  for  an 
optimizing  test  strategy  for  T&E  of  ALGM  programs  to  be  used  by  the 
program  manager  in  his  management  duties.   In  addition  to  providing  a 
test  strategy,  the  thesis  is  intended  to  be  a  self-contained  document 
wherein  the  ALGM  program  manager  has  the  T&E  policy,  requirements,  in- 
structions, organizations,  facilities,  potential  problem  areas,  test- 
ing requirements  and  typical  test  program  scope  identified.   This 
should  help  the  program  manager  resolve  some  of  the  T&E  management 
issues  and  help  him  avoid  some  of  the  problems  inherent  in  the  T&E 
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efforts.   The  test  strategy  will  aid  the  program  manager  in  the  budget- 
ing process,  in  his  early  program  planning,  in  bringing  T&E  issues 
into  focus,  and  will  serve  as  a  source  material  in  preparation  of  such 
program  documents  as  Decision  Coordinating  Papers  and  Test  and  Evalua- 
tion Master  Plans. 

C .   SCOPE 

The  thesis  addresses  T&E  from  general  policy  within  DoD  to  Navy 
application  and  specific  Navy  ALGM  program  implementation.   By  the 
nature  of  the  subject  matter  (management  of  T&E) ,  many  of  the  manage- 
ment issues  and  problems  are  broadly  applicable  and  relate  to  the  total 
spectrum  of  T&E.   However,  the  subject  can  be  applied  equally  to  ALGM 
defense  weapon  systems. 

The  thesis  investigates  management  of  T&E  in  reference  to  Navy 
Acquisition  Category  I  and  II  ALGM  programs.    The  program  life  cycle 
period  addressed  is  from  program  initiation  through  release  to  produc- 
tion (that  period  covered  by  Research,  Development,  Test  and  Evaluation 
(RDT&E)  funding) . 

An  Air-Launched  Guided  Missile  system  may  be  a  constituent  of  a 
larger  defense  weapon  system  which  has  as  its  primary  sub-system  a 
missile,  aircraft  avionics  sub-system,  and  support  equipment.   The 
missile  is  dominant  in  the  system  and  is  in  the  center  of  most  T&E 
activity.   The  missile  is  constructed  of  sections,  these  typically  be- 
ing guidance,  control,  warhead  and  propulsion.   Within  each  missile 


The  Navy  Acquisition  Categories  (ACAT)  are  defined  in  reference  4. 
ACAT  I  is  for  systems  with  dollar  thresholds  of  $75  M  RDT&E  and  $300  M 
production,  and  ACAT  II  have  $20  M  RDT&E  and  $50  M  production 
thresholds. 
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section  are  components,  assemblies,  and  parts.   ALGMs  currently  within 
the  Navy  inventory  are  comprised  of  two  broad  categories  or  groups, 
air-to-air  and  air-to-surface.   Typical  within  the  air-to-air  group 
are  Sidewinder,  Sparrow,  and  Phoenix.   The  air-to-surface  group  includes 
Condor,  Shrike,  Standard  Arm,  and  HARM. 

D.  ISSUES 

The  major  issue  addressed  in  this  thesis  is  how  can  T&E  be  managed 
effectively  and  efficiently  for  ALGM  programs.   Related  issues  include 
whether  there  is  an  optimal  test  strategy  which  minimizes  proliferation 
of  requirements ,  resources,  and  facilities,  yet  provides  resolution  to 
development  test  questions  and  provides  significant  and  relevant  infor- 
mation to  support  milestone  decisions.   Also,  can  the  problems  inherent 
to  T&E  be  reduced  by  implementation  and  procedural  changes  or  adjust- 
ments?  Specific  areas  of  interest  include:   what  T&E  is  required,  de- 
sired, and  affordable;   who  performs  the  T&E  and  when  is  it  accomplished; 
what  is  the  best  approach  to  implement  to  satisfy  the  ALGiM  T&E  require- 
ments, yet  is  within  program  dollar  and  schedule  constraints? 

These  and  similar  issues  relating  to  the  management  of  an  ALGM  pro- 
gram's T&E  activities  are  continually  addressed  in  the  thesis  and 
potential  solutions  are  explored. 

E.  RESEARCH  METHOD 

A  data  and  literature  search  was  performed  initially,  and  all  perti- 
nent DoD  directives,  Navy  instructions,  and  other  T&E  guidelines  reviewed. 
T&E  facilities  and  organizations  which  might  have  a  role  in  an  ALGM 
program  were  examined. 

Interviews  were  conducted  with  participants  in  ALGM  T&E  efforts 
(Appendix  A) ,  both  those  in  upper  level  management  and  review  cycles 
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and  those  who  directly  implement  program  T&E  activities.   Activities 
interviewed  include: 

1.  Deputy  Director  of  Defense  Research  and  Engineering,  Test  and 
Evaluation  (DD(TSE)) 

2.  Offices  of  the  Chief  of  Naval  Operations  (OPNAV) 

3.  Naval  Material  Command  (NAVMAT) 

4.  Naval  Air  Systems  Command  (NAVAIR) 

5.  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 

6.  Navy  Laboratories 

7.  Participating  Field  Activities 

8.  Development  Contractors 
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II.   TEST  AND  EVALUATION  AND  THE  SYSTEM  LIFE  CYCLE 

A.   NATURE  AND  PURPOSE  OF  T&E 

The  terns  "test"  and  "evaluation"  symbolize  distinguishable  func- 
tions in  the  RDT&E  process.   "Test"  denotes  the  actual  testing  of  hard- 
ware/software (models,  prototypes,  production  equipment,  computer  pro- 
grams) to  obtain  data  valuable  in  developing  new  capabilities,  managing 
the  developing  activities,  or  making  decisions  at  program  milestones 
or  on  the  allocation  of  resources.   "Evaluation"  denotes  the  process 
whereby  the  information  content  in  the  data  is  logically  assembled  and 
analyzed  to  aid  in  making  systematic  decisions.   In  a  broad  sense,  T&E 
may  be  defined  as  the  physical  testing,  experimentation  and  analyses 
performed  during  the  course  of  research,  development,  introduction  and 
employment  of  a  weapon  system  or  sub-system,  and  the  analytical  or 
evaluative  studies  performed  using  the  data  generated  [Ref .  5] . 

T&E  is  an  integral  part  of  all  phases  of  the  development  of  systems 
and  equipments  and  provides  information  for  a  number  of  purposes  and 
several  different  classes  of  information  users.   T&E  provides  the 
following  to  the  users  and  developers  of  systems; 

1.  Information  for  development 

2.  Information  for  acquisition  milestone  decisions 

3.  Information  for  effective  operational  utilization 

4.  Information  for  assessment  of  acquisition  risks 
These  are  discussed  briefly. 

1.   Information  for  Development 

Testing  of  systems  under  development  is  an  inherent  part  of  the 
Research  and  Development  (R&D)  process  through  which  technical 
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deficiencies  and  uncertainties,  as  well  as  design  and  operation  prob- 
lems are  identified  and  resolved.   This  testing  provides  design  data 
feedback  to  the  developing  agency  and  development  contractors.   The 
data  feedback  is  also  useful  in  the  design-test-evaluate-redesign  pro- 
cess which  is  basic  to  the  development  of  reliable  systems  [Ref .  5] . 

2.  Information  for  Acquisition  Milestone  Decisions 

The  major  milestone  decisions  in  the  development  of  a  system, 
to  initiate  advanced  development,  conduct  full-scale  development,  and 
produce  a  system,  are  by  nature  investment  decisions.   The  issue  at 
these  milestone  decision  points  is  whether  initiating  or  continuing  the 
acquisition  will  result  in  the  most  productive  use  of  the  resources 
(money,  material,  and  personnel) .   T&E  helps  provide  the  basis  for 
these  decisions,  and  the  best  information  possible  must  be  obtained  to 
aid  the  decision  process.   In  DoD  Directive  5000.3  the  statement  is 
made  that  the  basis  for  decisions  to  commit  added  resources  to  a  pro- 
gram shall  be  successful  accomplishment  of  the  T&E  objectives  [Refs. 
5  and  6] . 

The  Deputy  Director  (Test  and  Evaluation) ,  Office  of  the 
Director  of  Defense  Research  and  Engineering,  in  his  overview  statement 
to  Congress  in  March  1977,  stated  that  the  primary  purpose  of  T&E  is  to 
provide  information  for  program- level  decision  making.   Two  important 
questions  addressed,  using  T&E  information  as  a  basis,  are  "How  well 
current  objectives  are  being  met  and  what  is  likely  to  be  the  ultimate 
outcome  of  the  program."   Note,  one  question  deals  with  the  past  and 
the  other  with  the  future  [Ref.  3] . 

3 .  Information  for  Effective  Operational  Utilization 
Information  obtained  both  through  development  and  operational 

testing  provides  a  valuable  data  base  to  the  operational  user.   Typical 
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is  data  obtained  by  the  exploration  of  the  performance  envelopes  of 
the  system.   With  this  information,  tactics  can  be  developed  and 
doctrine  established  for  the  most  effective  utilization  of  the  system. 
4.   Information  for  Assessment  of  Acquisition  Risks 

Unfortunately,  many  times  T&E  does  not  provide  information 
which  leads  to  clear-cut  decisions.   The  T&E  information  does,  however, 
provide  the  decision-maker  with  a  means  to  make  judgments  and  to  assess 
the  associated  program  risks,  whether  they  be  technical,  managerial, 
cost  or  schedule.   DoDD  5000.3  states  directly  that  T&E  shall  be  con- 
ducted as  necessary  to  provide  the  information  to  assess  the  programs 
acquisition  risks  [Ref .  6] . 

B.   DOD  POLICY  FOR  T&E 

The  current  DoD  policy  for  T&E  has  evolved  from  the  changes  and 
direction  initiated  in  1969  by  David  Packard,  Deputy  Secretary  of 
Defense,  when  he  began  to  institute  reforms  in  the  system  acquisition 
process.   Among  these  reforms  was  that  needs  and  proposed  solutions 
should  be  validated,  preferably  by  experimental  verification  (testing) . 
Mr.  Packard  proposed  early  demonstration  in  areas  possessing  high 
development  risk,  emphasizing  T&E  as  being  of  paramount  importance 
[Ref.  7].   A  study  of  the  major  weapon  system  acquisition  process, 
also  in  1969,  by  a  Blue  Ribbon  Defense  Panel  established  by  the  Presi- 
dent and  Secretary  of  Defense  included  the  subject  of  T&E  [Ref.  8] . 
This  study  concentrated,  however,  on  OT&E  and  essentially  found  it  to 

be  unsatisfactory,  recommended  changes  to  make  it  more  effective,  and 

2 
concurred  with  its  significance  in  the  system  acquisition  process. 


2 
For  a  more  detailed  account  of  this  study,  as  well  as  other  studies 

on  T&E  and  the  role  of  OT&E  in  the  system  acquisition  process,  see 

thesis  by  W.  C.  Bowes  [Ref.  9] . 
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From  this  study  and  the  implementation  of  Mr.  Packard's  philosophy, 

DoD  Directive  5000.1  was  issued  in  1971  containing  requirements  for 

acquisition  of  major  defense  systems.   Addressing  T&E  policy, 

DoDD  5000.1  stated: 

"Test  and  evaluation  shall  commence  as  early  as  possible.   A 
determination  of  operational  suitability,  including  logistic  support 
requirements,  will  be  made  prior  to  large-scale  production  commit- 
ments, making  use  of  the  most  realistic  test  environment  possible 
and  the  best  representation  of  the  future  operational  system  avail- 
able.  The  results  of  this  operational  testing  will  be  evaluated 
and  presented  to  the  DSARC  at  the  time  of  the  production  decision" 
[Ref.   1]. 

DoD  then  responded  with  DoD  Directive  5000.3  in  197  3  [Ref.  2]  which 

contained  the  requirements  for  test  and  evaluation. 

A  similar  statement  regarding  T&E  policy  is  contained  in  the 
January  1977  revision  of  DoDD  5000.1  [Ref.  10],   Thus  this  directive 
provides  the  basic  foundation  for  T&E  as  it  is  implemented  today. 

DoD  Directive  5000.3  establishes  the  policy  for  the  conduct  of  test 
and  evaluation  by  the  Military  Departments  and  Defense  Agencies  (DoD 
Components)  in  the  acquisition  of  defense  systems.   This  document  con- 
tains general  policies  and  principles  for  T&E.   It  states  that  T&E 
shall  be  commenced  as  early  as  possible  and  conducted  throughout  the 
system  acquisition  process,  and  that  acquisition  schedules  will  be 
based  upon  accomplishing  T&E  milestones  prior  to  decision-making. 
Thus  the  T&E  program  is  planned  and  executed  concurrently  with  the 
development  phases  to  support  the  various  users  of  the  T&E  information. 
DoDD  5000.3  covers  the  requirements  for  T&E,  its  function  and  its  role. 
It  also  specifies  the  requirement  for  the  generation  of  an  overall  T&E 
plan  and  contains  procedures  and  requirements  for  T&E  inputs  to  the 
major  program  review  with  the  Defense  System  Acquisition  Review  Council 
(DSARC)  and  into  the  program's  Decision  Coordinating  Paper  (DCP) .   Of 
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importance  to  note  is  that  the  initial  DCP  presented  to  the  DSARC  at 
Milestone  I  will  identify  the  critical  questions  and  areas  of  risk  to 
be  resolved  by  T&E,  and  provide  test  objectives,  schedules  and  mile- 
stones.  In  addition,  revised  DCPs  at  later  milestones  will  give  re- 
sults of  T&E  accomplished  to  date  and  update  all  other  T&E  areas.   The 
DSARC  at  major  milestones  is  charged  with  the  responsibility  to  assess 
and  comment  to  SECDEF  as  to  the  adequacy  of  the  test  results  to  support 
a  decision  to  proceed  with  development.   The  DSARC  input  on  T&E  is 
obtained  from  the  Office  of  the  Deputy  Director  (Test  and  Evaluation) 
whose  function  is  to  oversee  defense  T&E  activities  and  set  policy  for 
conduct  of  T&E  [Ref .  6] . 

DoDD  5000.3  also  contains  policy  for  T&E  planning  in  DoD.   It  states: 

"The  DoD  Component  will  prepare  as  early  as  possible  in  the 
acquisition  process,  and  prior  to  initiation  of  Full-Scale  Develop- 
ment an  overall  test  and  evaluation  plan  to  identify  and  integrate 
the  effort  and  schedules  of  all  T&E  to  be  accomplished  and  to 
insure  that  all  necessary  T&E  is  accomplished  prior  to  the  key 
decision  points.   The  TEMP  (T&E  Master  Plan)  will  be  kept  current 
by  the  DoD  Component"  [Ref.  6] . 

A  significant  feature  of  the  T&E  policy  promulgated  by  DoDD  5000.1 

and  5000.3  is  the  increased  importance  and  strength  of  the  independent 

test  agency.   This  evolved  in  response  to  the  previously  cited  Blue 

Ribbon  Defense  Panel  study  and  was  concurred  with  by  the  study  conducted 

by  the  Commission  on  Government  Procurement  [Refs.  8  and  11]. 

C.   RELATIONSHIP  TO  SYSTEM  LIFE  CYCLE 

Figure  1  depicts  the  system  life  cycle  as  adapted  from  Ref.  7.   The 
process  begins  with  requirements  or  needs  and  evolves  through  the 
development  phases  to  production  and  operational  use.   Figure  2  shows 
the  T&E  areas  corresponding  to  the  life  cycle  phase  for  defense  acquisi- 
tion systems  according  to  T&E  types.   Also  shown  are  the  major  program 
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Figure  1.   System  Life  Cycle  (Adapted  From  Ref.  ?) 
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milestone  decision  points.   T&E  plays  a  vital  part  in  Milestones  I,  II 
and  III.   Milestone  0  is  the  program  initiation  point  where  a  mission 
need  has  been  identified  and  exploration  of  alternative  solutions 
authorized.   Milestone  I  is  the  program  validation  decision  point  where 
selected  alternatives  continue  into  a  demonstration  and  validation 
effort.   A  DCP  is  required  at  this  time,  containing,  among  other  things, 
identified  T&E  objectives  and  issues.   Milestone  II  is  the  point  where 
a  commitment  to  full-scale  development  is  made  for  the  preferred  system. 
Demonstration  and  validation  of  T&E  objectives  should  have  been  com- 
pleted and  the  results  should  support  this  decision.   The  DCP  is  updated 
appropriately.   Milestone  III  is  the  point  where  the  decision  is  made  to 
produce  and  deploy  the  system.   T&E  makes  a  major  input  at  this  program 
milestone  point  [Refs.  4,  10  and  12] . 

There  are  three  types  of  T&E  used  in  the  system  life  cycle: 

1.  Development  Test  and  Evaluation  (DT&E) 

2.  Operational  Test  and  Evaluation  (OT&E)  which  is  composed  of: 

a.  Initial  OT&E  (IOT&E) 

b.  Follow-on  OT&E    (FOT&E) 

3.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 
References  4  and  6  describe  in  detail  the  three  types  of  T&E. 

Appendix  B  contains  a  brief  description  of  each  type  for  quick  reference. 

D.   PROGRAM  REVIEWS 

As  noted  in  section  II  B,  a  principal  purpose  of  T&E  is  to  provide 
assistance  to  decision-makers  at  program  reviews  or  milestones  by  pro- 
viding data  and  information.   Other  important  program  reviews,  keyed 
especially  to  the  commitment  of  resources,  rely  much  on  T&E  information. 
Examples  are  Congressional  reviews  with  the  House  and  Senate  Armed 
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Services  and  Appropriations  Committees.   At  these  reviews,  T&E  results, 
in  addition  to  other  program  inputs,  are  used  to  determine  the  commit- 
ment of  funds,  program  continuation  or  program  alteration.   For  major 
programs,  reviews  occur  at  various  upper  management  levels  throughout 
the  organization,  and  at  which  T&E  results  are  vital. 

In  addition  to  upper  level  program  reviews,  T&E  plays  an  important 
role  in  reviews  held  throughout  the  system  life  cycle  at  the  develop- 
ment organization  level.   Design  reviews  occur  as  necessary  during  the 
acquisition  process,  involving  the  development  agency,  the  development 
contractor,  and  the  applicable  T&E  agencies  and  activities.   Testing 
data  and  results  are  an  inherent  and  important  part  of  these  reviews, 
which  provide  valuable  design  and  development  information.   Also  occur- 
ring at  the  development  organization  level  are  periodic  reviews  of  the 
T&E  program  to  examine  test  progress  and  for  review  of  problem  identi- 
fication and  resolution  information. 


26 


III.   NAVY  TEST  AND  EVALUATION  REQUIREMENTS 
AND  PROGRAM  STRUCTURE 

This  section  contains  the  Navy's  implementation  of  the  DoD  direc- 
tives for  T&E  and,  where  appropriate,  presents  unique  features  relative 
to  ALGM  programs . 

A.   NAVY  POLICY  FOR  T&E 

The  Navy's  most  recent  implementation  response  to  DoDD  5000.3  was 
OPNAVINST  3960.10,  issued  in  October  1975,  which  established  the  policy 
for  T&E  in  Navy  acquisition  programs.   This  is  the  principal  Navy  policy 
document  for  T&E.   OPNAVINST  3960.10  defines  the  T&E  responsibilities 
of  Navy  activities  and  indicates  procedures  for  planning,  conducting, 
and  reporting  T&E.  It  delineates  the  complementary  relationship  between 
DT&E  and  OT&E,  and  establishes  procedures  and  format  requirements  for 
Test  and  Evaluation  Master  Plans  (TEMPs)  [Ref .  4] .   The  TEMP  is  the 
controlling  management  document  which  defines  T&E  for  each  acquisition 
program.   OPNAVINST  3960.10  must  be  understood  thoroughly  by  the  program 
manager,  his  designee  for  T&E,  and  others  responsible  for  implementing 
T&E  policy  on  a  program. 

OPNAVINST  3960.10  also  clarifies  the  responsibility  of  the  indepen- 
dent test  agency.   The  agency  for  the  Navy,  OPTEVFOR,  and  its  commander, 
COMOPTEVFOR,  has  been  assigned  this  responsibility. 

The  Department  of  Navy  RDT&E  Management  Guide  [Ref.  5]  also  provides 
a  valuable  guideline  on  management  of  Navy  T&E.   Chapter  VII  and 
Appendix  G  of  Ref.  5  are  devoted  totally  to  Navy  T&E  and  provide 
appropriate  amplification. 
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B.   MANAGEMENT  PHILOSOPHY  AND  ORGANIZATION 

The  requirement  for  management  of  a  Navy  program's  T&E  efforts 
rests  primarily  with  the  Program  Manager.   Even  though  a  broad  organi- 
zation exists  in  Navy  T&E,  the  Program  Manager  is  responsible  for 
planning,  implementing  and  reporting  the  T&E  for  the  weapon  system. 
OPNAVINST  3960.10  points  out  that  the  Program  Manager  within  the 
Developing  Agency  is  responsible  for  development  and  execution  of  an 
adequate  T&E  program.   His  T&E  responsibilities  include  defining  a 
test  program  which  illuminates  test  issues  and  problems;  preparation 
and  updating  of  the  TEMP;  and  arranging  for  performance  of  the  required 
and  planned  testing  and  subsequent  evaluation  [Ref .  4] . 

In  the  conduct  of  the  T&E  program,  however,  OPNAVINST  3960.10 
delegates  authority  for  the  three  types  of  testing  to  two  different 
organizations.   The  Developing  Agency  is  responsible  for  DT&E.   For 
ALGMs,  the  Developing  Agency  is  also  responsible  for  PAT&E.   This  in- 
cludes planning,  conducting  and  reporting.   In  addition  the  Developing 
Agency  maintains  liaison  with  COMOPTEVFOR  concerning  the  DT&E  program 
and  provides  COMOPTEVFOR  with  appropriate  results.   For  OT&E,  the  Navy 
has  assigned  the  responsibility  to  OPTEVFOR,  which  is  an  organization 
separate  and  distinct  from  the  developing  and  procuring  command,  and 
from  the  using  command  (although  attached  to  CNO) .   OT&E  is  planned  and 
conducted  by  OPTEVFOR,  and  results  are  reported  directly  to  the  CNO 
by  COMOPTEVFOR.   In  addition,  OPTEVFOR  provides  the  Developing  Agency 
with  all  significant  OT&E  results  [Ref.  4] . 

An  organizational  chart  which  depicts  the  Navy's  T&E  agencies 
(along  with  that  at  the  SECDEF  level)  is  shown  in  Figure  3.   It  also 
shows  those  field  activities  involved  in  ALGM  program.   Descriptions 
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of  each  organization's  functions  and  responsibilities  can  be  found  in 
Ref.  5. 

Although  complementary,  DT&E  and  OT&E  are  indeed  separate  and  dis- 
tinct, both  in  their  basic  goals  and  objectives  and  in  their  implemen- 
tation and  conduct.   From  their  requirements  foundation,  DT&E  works  to 
specifications  stated  in  functional  or  technical  terms  whereas  OT&E 
attempts  to  state  requirements  in  operational  terms  (Appendix  B) .   Thus, 
their  criteria  thresholds  are  generally  also  different.   Many  times  in 
DT&E,  tests  concern  only  the  weapon  or  a  part  thereof  whereas  in  OT&E, 
tests  primarily  are  concerned  with  the  complete  weapon  system.   Another 
difference  is  that  DT&E  often  is  testing  and  measuring  a  specific  para- 
meter, holding  other  things  constant  to  see  the  effect,  whereas  in 
OT&E  many  times  no  specific  parameter  is  identified  and  only  an  opera- 
tional environment  is  created  to  see  what  the  test  results  indicate. 
Thus,  repeatability  of  tests  is  usually  possible  in  DT&E  whereas  in 
OT&E  it  is  not.   Further,  the  people  involved  in  DT&E  are  usually  tech- 
nical whereas  those  in  OT&E  are  not  technical  but  operational.   RAdm 
Monroe,  former  COMOPTEVFOR,  addressed  these  differences  in  his  comments 
about  OT&E  stating  "that  DT&E  is  a  science  and  OT&E  an  art,  and  if 
there  is  duplication  between  the  two,  T&E  is  not  being  properly 
planned"  [Ref.  13] . 

On  paper,  the  success  of  a  development  program's  T&E  in  the  Navy 
rests  with  the  Program  Manager.   He  is  responsible  for  the  coordination 
of  the  various  parts  of  the  T&E  effort,  including  the  specific  imple- 
mentation of  the  many  facets  of  the  DT&E  program.   The  program's  success 
is  also  dependent  on  the  Program  Manager's  relationship  and  working 
procedures  with  OPTEVFOR,  and  on  his  interfaces  with  organizations 
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which  have  Navy  T&E  responsibilities.   In  the  exercise  of  his  respon- 
sibilities, the  Program  Manager  must  deal  with  the  contractors,  par- 
ticipating field  activities,  and  various  testing  agencies  to  implement 
the  DT&E  program.   He  must  operate  within  the  matrix  organization  of 
the  Navy  for  his  technical  assistance  and  support.   And  he  must  nego- 
tiate with  OPTEVFOR  to  obtain  a  reasonable  and  affordable  OT&E  program, 
yet  meet  OT&E  requirements.   Those  in  the  DT&E  program  are  directly 
dependent  on  the  Program  Manager  and  OPTEVFOR  is  indirectly  dependent 
on  him  as  he  provides  the  funds  to  support  both  DT&E  and  most  of  OT&E. 
In  addition  the  Program  Manager  must  work  with  System  Command  and 
NAVMAT  functional  groups  on  test  coordination  and  facilities  planning, 
and  his  program  must  sustain  the  reviews  and  scrutinizing  of  CNO  and 
DD(T&E)  . 

Two  especially  difficult  situations  confront  the  Program  Manager  in 
his  management  responsibility  and  structure.   First,  it  is  the  position 
of  DD(T&E)  that  more  value  should  be  placed  on  OPTEVFOR  information  for 
milestone  decision-making  than  on  the  developing  agency's  data  because 
OPTEVFOR  is  an  independent  test  agency.   DD(T&E)  feels  that  the  develop- 
ing agency  has  a  vested  interest  in  the  program,  therefore  possesses  an 
advocacy  position.   Thus   DD(T&E)  judges  data  received  from  DT&E  in 
this  light.   Bowes  made  a  similar  observation  in  his  thesis  [Ref .  9] . 

Second,  COMOPTEVFOR  is  double-hatted,  heading-up  OPTEVFOR  and  the 
T&E  division  in  CNO  (OP-983) .   He  also  acts  as  the  Assistant  Director 
for  OT&E  (OP-098C)  in  support  of  the  Director  RDT&E  in  CNO,  reporting 
from  this  position  directly  to  the  CNO.   In  the  Program  Manager's  dis- 
cussions and  negotiation  with  OPTEVFOR,  the  Program  Manager  is  operating 
from  a  potentially  difficult  position.   COMOPTEVFOR ' s  position  in  the 
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Navy  organizational  structure  is  very  powerful  compared  to  that  of  a 
Program  Manager  because  of  their  relative  positions  with  respect  to 
CNO. 

C.   TYPES  AND  CATEGORIES  OF  TESTS 

Figure  2  depicted  the  TSE  types  and  categories  of  tests.   Each  of 
the  T&E  categories  (phases)  will  be  discussed  separately  as  each  one 
possesses  unique  objectives  and  features  of  T&E.   Much  of  the  informa- 
tion can  be  found  in  OPNAVINST  3960.10.   It  is  presented  here  because 
it  is  continually  referred  to  throughout  this  thesis. 

1.   DT&E 

a.  DT-I 

Development  contractors  and/or  participating  field  activi- 
ties perform  that  DT&E  possible  at  this  time  during  the  conceptual 
phase  with  experimental  hardware  to  demonstrate  concepts  and  feasible 
approaches  to  support  the  program  validation  decision  (Milestone  I) . 

b.  DT-II 

Development  contractors  and/or  participating  field  activi- 
ties conduct  this  DT&E  during  the  validation  phase,  normally  at  the 
major  component/sub- sus tern  level,  to  support  the  full-scale  develop- 
ment decision  (Milestone  II) .   The  hardware  is  usually  of  advanced 
development  quality  and  its  purpose  is  to  demonstrate  that  design  risks 
have  been  identified  and  minimized. 

c.  DT-III 

Development  contractors  design,  build,  and  test  engineering 
development,  prototype,  and  pilot  or  limited  production  hardware.   They 
essentially  perform  design  evaluation  testing,  including  redesign  and 
re- test  as  necessary,  to  demonstrate  compliance  with  contractual 
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(specification)  requirements.   Testing  areas  include,  as  a  minimum, 
performance,  environmental,  reliability /maintainability,  compatibility, 
and  safety.   Tests  are  both  sub-system  and  system  level  (ground, 
captive  and  free  fight)  with  environmental  evaluation  tests,  test 
analyze  and  fix,  design  proof  tests,  and  qualification  test  programs. 
Reliability  and  maintainability  demonstration  programs  are  also 
generally  conducted. 

The  Program  Manager  may  designate  a  participating  field 
activity  as  the  Technical  Manager  to  provide  necessary  specialized 
technical  and  support  engineering  skills  during  this  hardware  develop- 
ment period.   In  this  capacity  the  participating  field  activity  may 
witness  contractor  performed  DT&E  and  may  choose  to  perform  (repeat) 
certain  tests  judged  to  be  critical  to  program  success.   In  addition, 
certain  components  of  ALGMs  (e.g.,  warheads,  fuzes,  and  propulsion 
systems)  are  designed,  developed  and  tested  by  other  participating 
field  activities  which  have  special  competence,  experience,  and/or 
facilities  necessary  for  conduct  of  these  sometimes  hazardous  develop- 
ment efforts. 

Thus,  contractually  and/or  through  task  assignments  to 
participating  field  activities,  design,  analysis,  test  and  evaluation 
proceeds  through  DT-III,  including  formal  demonstration  and  qualifica- 
tion testing,  sometimes  referred  to  as  Contractor  Technical  Evaluation 
or  Contractor  Demonstration  Tests.   The  final  sub-phase  of  DT-III  for 
most  ALGMs  consists  of  subjection  to  a  Navy  Technical  Evaluation, 
usually  referred  to  as  TECHEVAL.   Its  purpose  is  to  provide  evidence 
to  the  Program  Manager  for  certification  of  readiness  for  OPEVAL 
(operational  evaluation) .   The  TECHEVALs  for  ALGMs  have  historically 
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been  performed  for  the  Program  Manager  and  NAVAIR  by  a  field  activity 
other  than  the  one  designated  as  Technical  Manager.   This  has  usually 
resulted  in  PMTC,  Point  Mugu  performing  this  function.   TECHEVALs  per- 
formed by  Point  Mugu  previously  have  placed  significant  emphasis  on 
environmental  testing  in  excess  of  ALGM  specification  limits  and 
ground  and  airborne  reliability  testing  to  demonstrate  compatibility 
with  specification  numeric  and  confidence  level  requirements.   Compli- 
ance with  maintainability  requirements  is  also  evaluated. 

DT-III  then  supports  the  first  major  production  decision 
(Milestone  III)  and  demonstrates  that  the  ALGM  design  meets  its  speci- 
fications in  performance,  reliability,  maintainability,  supportability, 
survivability,  system  safety,  and  the  totality  of  elements  of  electro- 
magnetic vulnerability.   In  addition,  operability,  human  factors, 
logistics  and  system  interface  compatibility  are  evaluated. 

d.   DT-IV 

This  phase  of  DT&E  is  conducted  after  the  first  major  pro- 
duction decision  to  verify  that  product  improvements,  or  correction  of 
design  deficiencies  discovered  during  OPEVAL,  FOT&E,  or  fleet  employ- 
ment, are  effective. 
2.   IOT&Z 

a.   OT-I 

IOT&E  commences  early  in  the  program  upon  the  request  of 
the  Program  Manager  with  OT-I  conducted  by  OPTEVFOR  during  the  con- 
ceptual phase  to  support  the  program  validation  decision  (Milestone  I). 
Most  acquisition  programs  do  not  require  OT-I.   However,  this  is  the 
time  for  OT&E  planning  to  take  place. 
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b.  OT-II 

OPTEVFOR  conducts  this  phase  of  IOT&E  during  the  validation 
phase  to  provide  an  early  estimate  of  projected  operational  effective- 
ness and  suitability  of  the  ALGM,  to  estimate  program  progress  and  ini- 
tiate development  of  tactics  and  finally,  to  identify  operational  issues 
for  consideration  during  the  full-scale  development  phase  (OT-III) . 
OPTEVFOR  will  participate  in  the  ALGM  captive  flight  test  program  to 
the  extent  practicable  and  may  also  monitor  tests  at  contractor  and  par- 
ticipating field  activities  facilities  to  aid  in  performance  of  its 
early  operational  assessment. 

c.  OT-III 

OPTEVFOR  conducts  this  phase  of  IOT&E  during  full-scale  de- 
velopment to  support  the  program's  first  major  production  decision 
(Milestone  III)  .   OT-III  concludes  with  a  final  sub-phase  identified 
as  OPEVAL  which  normally  uses  pilot  production  hardware  and  begins 
shortly  after  TECHEVAL  or  after  incorporation  of  changes  necessary  to 
correct  significant  problems  identified  by  TECHEVAL.   Prior  to  OPEVAL 
and  availability  of  pilot  production  hardware,  OPTEVFOR  will  continue 
its  IOT&E  participation  in  the  prototype  ALGM  evaluation  of  development 
hardware  with  captive  flight  tests  and  missile  launches  according  to 
the  program  test  plan.   The  purpose  of  OT-III  and  particularly  OPEVAL 
is  to  evaluate  ALGM  operational  effectiveness  and  suitability  (includ- 
ing availability,  compatibility,  interoperability,  reliability,  main- 
tainability, human  factors,  logistic  supportability  and  training 
requirements)  and  to  continue  tactics  development. 
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3.   FOT&E 

a.  OT-IV 

OPTEVFOR  may  conduct  this  phase  of  FOT&E  after  the  first 
major  production  decision  but  before  production  ALGMs  are  available  if 
hardware  problem  fixes  are  to  be  evaluated,  to  continue  tactics  develop- 
ment, or  to  complete  deferred  or  incomplete  IOT&E.   This  phase  may  also 
be  utilized  for  conduct  of  a  fleet  reliability  evaluation  program  bv 
OPTEVFOR  (i.e.,  early  introduction  and  evaluation  in  actual  fleet  en- 
vironment) . 

b .  OT-V 

OPTEVFOR  conducts  this  FOT&E  on  production  svstems  as  soon 
as  they  are  available.   Objectives  include  demonstration  of  the  achieve- 
ment of  program  objectives  for  production  system  operational  effective- 
ness and  operational  suitability,  OT&E  of  the  system  in  new  environments, 
or  in  new  applications,  or  against  new  threats. 

Figures  4  and  5  show  the  scope  of  DT&E  and  OT&E  for  ALGMs 
in  a  different  format.   Depicted  are  test  categories  and  sub-categories 
for  DT&E  and  OT&E,  and  typical  phases  in  which  the  tests  are  performed. 

D.   RESPONSIBILITY  FOR  TESTS 

DT&E  is  planned  by,  conducted  by  or  for,  monitored  by,  and  reported 
by  the  developing  agency.   For  ALGM  programs,  the  Program  Manager 
operates  from  the  developing  agency  organization  (NAVAIR)  and  delegates 
responsibility  to  support  groups,  field  activities  and  contractors  for 
various  DT&E  tests.   The  delegation  vehicles  are  task  assignments  or 
contracts  as  appropriate  and  acceptance  by  the  agencies  constitutes 
obligation. 
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OT&E  is  planned  by,  conducted  by  or  for,  and  reported  directly  to 
C1IO  by  COMOPTEVFOR.   With  only  a  relatively  modest  number  of  personnel 
and  resources,  COMOPTEVFOR  relies  heavily  on  the  facilities,  resources, 
and  personnel  of  the  operating  forces  and  government  field  activities 
for  carrying  out  his  mission.   He  exercises  operational  control  over 
fleet  units  assigned  for  project  support.   Close  liaison  is  authorized 
and  exercised  with  appropriate  elements  of  NAVMAT,  the  System  Commands 
(1IAVAIR  for  ALGM)  and  other  T&E  organizations  to  facilitate  test  support 
and  information  flow  in  carrying  out  assigned  projects  [Ref .  5] . 

E.   CONDUCT  OF  TESTS  FOR  ALGMs 

For  DT&E  of  ALGMs,  tests  are  conducted  by  contractors,  Navy  Labora- 
tories, and  participating  field  activities.   OPTEVFOR  is  responsible 
for  the  conduct  of  the  OT&E  tests,  which  begin  during  the  validation 
phase  for  most  ALGM  programs.   Of  importance  to  note  is  the  planned 
overlap  and  concurrence  of  DT&E  and  OT&E  throughout  the  acquisition 
period. 

The  development  contractors  perform  much  of  the  DT&E  testing. 
Several  contractors  are  involved  during  the  conceptual  phase  when  the 
various  concepts  are  evaluated.   During  the  validation  phase,  the 
selected  contractors  conduct  ground  tests,  and  usually  with  government 
aid,  conduct  captive  flight  tests  and  flight  test  firings.   During 
full-scale  development,  an  ALGM  development  contractor  conducts  the 
DT&E  he  is  contractually  responsible  for,  which  includes  performance, 
environmental,  reliability  tests  (as  identified  in  section  III  C)  at 
the  sub-system  and  system  level,  on  the  ground  and  in  captive  flight 
tests  and  firings.   Various  contractors  responsible  for  sub-systems 
of  the  missile  system  will  perform  similar  applicable  DT&E. 
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Navy  development  laboratories  and  other  participating  field  activi- 
ties perform  DT&E  testing  in  ALGM  programs  as  well.   Development  labora- 
tories support  NAVAIR  and  the  Program  Manager  in  the  early  conceptual 
phase  and  usually  continue  in  this  capacity  throughout  the  development 
cycle.   During  DT-I  the  laboratories  conduct  conceptual  evaluations 
along  with  the  contractors.  During  DT-I I,  they  work  with  the  selected 
contractors  to  demonstrate  that  design  risks  have  been  identified  and 
the  DT&E  efforts  support  the  Milestone  II  decision.   Laboratories  play 
an  especially  important  role  in  the  conduct  of  captive  and  free-flight 
tests.   During  DT-III  they  monitor  much  of  the  contractor  testing  and 
continue  their  role  in  captive  and  free-flight  testing.   They  provide 
an  assessment  of  the  system  and  provide  DT&E  results  to  the  Program 
Manager  for  his  use  concerning  the  production  decision.   Participating 
field  activities  perform  DT&E  for  the  Program  Manager  in  certain  sub- 
system areas  which  require  special  skills  and/or  facilities,  namely  in 
warhead,  fuze  and  propulsion  areas.   PMTC,  Point  Mugu  has  historically 
conducted  the  TECHEVAL  for  ALGMs . 

OPTEVFOR  begins  conducting  tests  and  performing  evaluations  of  the 
ALGM  data  available  during  the  validation  phase's  activities.   This 
initial  testing  in  OT-II  is  in  conjunction  with  the  DT&E  program,  con- 
ducting some  of  the  captive  flight  tests  and  firings  and  monitoring  the 
contractor,  laboratory  and  participating  field  activity  testing.   This 
provides  the  necessary  information  for  OPTEVFOR  to  make  its  projected 
operational  effectiveness  and  suitability  assessment  for  Milestone  II. 
During  full-scale  development,  OPTEVFOR  continues  to  monitor  the  DT&E 
efforts  and  conducts  many  of  the  program's  captive  flights  and  missile 
firings.   The  OT&E  efforts  in  this  phase  culminates  in  a  formal  OPEVAL 
which  OPTEVFOR  performs  on  the  system  forming  the  basis  for  its 
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Milestone  III  recommendation.   OPTEVFOR  also  conducts  or  coordinates 
the  operational  testing  associated  with  FOT&E. 

F.  FACILITIES  FOR  TSE  OF  NAVY  ALGMs 

Navy  ALGM  programs  use  six  of  the  26  activities  designated  elements 
of  the  DoD  T&E  Facility  Base,  also  referred  to  as  the  Major  Range  and 
Test  Facility  (MRTFB) .   The  six  are  listed  below.   Associated  mission 
statements  as  obtained  from  the  Navy  RDT&E  Management  Guide  [Ref.  5]  are 
given  in  Appendix  C. 

1.  Naval  Ordnance  Missile  Test  Facility,  White  Sands,  New  Mexico 

2.  Naval  Air  Test  Center,  Patuxent  River,  Maryland 

3.  Pacific  Missile  Test  Center,  Point  Mugu,  California 

4.  Naval  Weapons  Center  Ranges,  China  Lake,  California 

5.  Naval  Weapons  Laboratory  Ranges,  Dahlgren,  Virginia 

6.  Naval  Explosive  Ordnance  Disposal  Facility,  Indian  Head,  Maryland 
The  Program  Manager  and  persons  responsible  for  implementation  of 

ALGM  TSE  projects  can  obtain  the  capabilities  of  these  applicable  T&E 
facilities  by  consulting  references  cited  in  Chapter  7,  Section  073  of 
Ref.  1.   The  key  factor  in  obtaining  use  of  test  ranges  and  other 
facilities  is  early  contact  with  cognizant  test  facilities  personnel. 
Early  liaison  will  assist  in  the  definition  of  a  practical  test  plan 
to  be  incorporated  in  the  TEMP,  and  will  allow  the  facility  the  lead 
time  required  to  provide  the  required  support. 

G.  FUNDING 

T&E  funding  responsibility  rests  with  the  Program  Manager.   The 
Program  Manager  must  plan,  program,  budget,  and  fund  the  cost  of  all 
resources  identified  in  the  approved  TEMP  for  all  T&E  from  initial 
development  through  production,  except  for  some  fleet  related  expenses. 
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This  includes  all  DT&E  costs  and  practically  all  OT&E  costs  (OPNAVINST 
3960.10  cites  the  minor  exceptions) .   Since  the  Program  Manager  must 
budget  for  the  OT&E  costs,  it  is  essential  that  COMOPTEVFOR  identify 
all  requirements  in  sufficient  time  to  integrate  these  into  the  program 
schedule  and  budget  cycle  [Ref .  4] . 

H.   PLANNING 

OPNAVINST  3960.10  implements  the  planning  policy  of  DoDD  5000.3  by 
specifying  various  requirements  with  regards  to  the  TEMP.   It  notes 
that  the  TEMP  defines  the  test  and  evaluation  efforts  for  each  acquisi- 
tion program  and,  as  such,  contains  the  integrated  requirements  of  the 
developing  agency  for  DT&E  and  COMOPTEVFOR  for  OT&E,  their  schedules 
and  resource  requirements.   It  further  states  that  the  TEMP  will  be  pre- 
pared early  in  each  new  acquisition  program,  and  approved  prior  to 
Milestone  I;  the  TEMP  will  be  prepared  by  the  development  agency  with 
the  OT&E  portion  prepared  by  COMOPTEVFOR;  and  that  the  development 
agency  ensures  the  TEMP  accurately  reflects  its  planned  approach  to 
orovide  the  necessary  T&E  to  solve  development  design  issues.   Approval 
of  the  TEMP  constitutes  CNO  direction  to  conduct  the  T&E  program  defined 
therein.   An  enclosure  to  OPNAVINST  3960.10  provides  specific  instruc- 
tions for  TEMP  preparation  [Ref.  4] . 

In  addition  to  the  requirements  of  the  TEMP,  Navy  T&E  planning  in- 
formation is  contained  in  the  Decision  Coordinating  Paper,  which  is 
prepared  early  in  the  system  acquisition  process.   The  Decision  Coordi- 
nating Paper  identifies  the  critical  questions  and  areas  of  risk  to  be 
resolved  by  T&E,  establishes  the  initial  goals  and  thresholds,  and  also 
includes  a  summary  statement  of  test  objectives,  schedules,  and  mile- 
stones.  These  are  updated  at  subsequent  revisions  of  the  Decision 
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Coordinating  Paper,  and  at  the  major  milestone  decision  points 
[Ref.  6]. 

The  Program  Manager  must  plan  his  T&E  efforts  early  because  it  con- 
stitutes a  significant  portion  of  his  program  budget.   A  representative 
budget  must  be  submitted  at  Milestone  I,  therefore  important  and  de- 
tailed program  and  T&E  planning  must  occur  during  the  conceptual  phase. 
Plans  summarized  in  the  T&E  section  of  the  DCP  lay  the  groundwork  for 
earliest  possible  commencement  of  T&E  and  can  influence  significantly 
the  program's  budget.   These  planning  inputs  generally  are  prepared 
some  time  before  the  TEMP  is  assembled  for  Milestone  I.   TEMPs  to  date 
contain  a  level  of  detail  and  commitment  which  has  caused  them  to  be 
somewhat  controversial  and  as  a  result  agreement  and  approval  do  not 
occur  until  well  into  the  development  program. 

The  budget  process  contains  the  impetus  to  accomplish  early  T&E 
planning.   In  addition,  the  Program  Manager  must  realize  that  this  early 
planning  enables  him  to  determine  the  scope,  asset  and  resource  require- 
ments of  his  T&E  program.   The  early  visibility  will  provide  the  Program 
Manager  with  valuable  management  insight  into  his  planned  T&E  effort. 

I.   NAVY  T&E  PROGRAM  REVIEWS 

Aside  from  the  program  reviews  cited  in  section  II-E  which  are  de- 
pendent on  T&E  results  for  basic  inputs,  program  reviews  of  T&E  are 
conducted  periodically  by  the  Program  Manager,  his  T&E  coordinator,  or 
designee  for  T&E.   These  reviews,  among  other  things,  determine  test 
program  status,  identify  problems  and  determine  corrective  action  status 
and  responsibilities.   Many  times  the  T&E  program  reviews  are  a  part  of 
the  system  design  reviews,  as  a  broad  spectrum  of  program  personnel  are 
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interested  in  the  program's  T&E  activities.  These  reviews  involve  the 
participation  of  all  T&E  agencies  responsible  for  ALGM  T&E  activities, 
including  contractors,  Navy  laboratories,  participating  field  activities 
and  OPTEVFOR.  However,  OPTEVFOR's  role  in  program  reviews  at  the  develop- 
ment level  is  limited,  in  keeping  with  its  position  as  the  independent 
test  agency.  T&E  reviews  help  the  T&E  activities  maintain  the  impetus 
and  pace  commensurate  with  the  ALGM  development  progress. 

J.   INFORMATION  FLOW  AND  REVIEW 

T&E  provides  the  information  for  decision-making.   Whether  it  is  de- 
sign evaluation  data,  operational  utilization  data,  or  risk  assessment 
data,  the  T&E  information  must  be  disseminated  to  users  to  be  useful. 
The  Program  Manager  is  aware  of  all  T&E  activity.   All  DT&E  is  authorized 
by  the  Program  Manager  and  reported  to  his  office.   The  only  exception  to 
this  is  when  development  contractors  perform  design  tests  as  part  of 
their  internal  development  efforts;  however,  when  the  Navy  requests  such 
data,  the  contractor  is  usually  cooperative  in  providing  it.   Although 
COMOPTEVFOR  reports  directly  to  CNO,  COMOPTEVFOR  provides  significant 
OT&E  test  data  and  analysis  to  the  Program  Manager.   Thus,  through  the 
program  office  the  T&E  information  is  available  or  provided  directly  to 
the  necessary  users. 

The  TEMP  has  been  a  key  document  to  help  in  the  dissemination  of  T&E 
information.   Its  coordinated  and  integrated  characteristic  provides  pro- 
gram personnel  with  visibility  as  to  when  tests  are  occurring,  who  is 
responsible,  and  provides  other  related  test  management  information. 
This  enables  the  T&E  participants  and  users  of  the  data  to  be  involved 
in  the  testing,  either  actively  or  in  a  monitoring  capacity,  and  to  be 
aware  of  the  availability  of  T&E  information. 
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The  Navy's  philosophy  of  attempting  to  bring  OPTEVFOR  into  the  pro- 
gram's T&E  efforts  during  DTSE  helps  greatly  in  an  early  assessment  of 
the  system  and  planning  for  OPEVAL.   The  results  from  DT&E  and  OT&E 
figure  prominently  in  the  development  of  the  publications,  manuals  and 
training  aids  for  the  operating  forces.   This  also  pertains  to  the 
logisticians  in  their  development  and  proofing  of  the  Integrated  Logis- 
tic Support  Program. 
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IV.   PROBLEMS  IN  THE  MANAGEMENT  OF  T&E 
AND  OF  CONCERN  TO  ALGM  PROGRAMS 

Problems  will  be  addressed  which  have  been  identified  throughout  the 
T&E  community  and  in  ALGM  programs,  and  of  which  the  ALGM  Program  Manager 
should  be  aware.   Some  of  the  problems  identified  are  of  such  a  nature 
that  their  application  can  be  to  many  program  types  or  to  T&E  in  general; 
however,  all  the  problems  alluded  to  should  be  a  specific  concern  to  an 
ALGM  Program  Manager,  his  designee  for  T&E,  or  any  other  individual  re- 
sponsible for  implementation  of  ALGM  T&E. 

The  problems  are  categorized  into  five  areas: 

A.  Planning 

B.  Proliferation  of  Requirements  and  Reviews 

C.  Resource  and  Schedule  Constraints 

D.  Redundant  Testing 

E.  Management  of  Diverse  Activities  and  Functions 

A.   PLANNING 

Probably  the  foremost  problem  relative  to  inadequate  and  inefficient 
T&E  is  poor  planning  by  those  responsible  for  performance  and  implemen- 
tation of  the  program.   Planning  problems  encompass  several  dimensions, 
some  of  the  important  ones  being: 

1.  Inadequate  and  uncoordinated  planning  early  in  the  development 
cycle  and  among  the  various  responsible  organizations 

2.  Implementation  of  "success  oriented"  planning 

3.  Lack  of  flexibility  in  T&E  planning  to  accommodate  problems  and 
unknowns 
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4.  Unrealistic  requirements  in  the  planning  efforts 

5.  Lack  of  design  of  tests,  definition  of  test  criteria,  and  know- 
ing the  use  of  the  test  data 

6.  Lack  of  definition  of  a  mission  profile 

7.  Difficulty  in  converting  operational  requirements  into  test 
specification  requirements 

8.  Inability  to  convert  specification  performance  requirements 
into  T&E  asset  (resource)  requirements 

These  problems  are  discussed  in  the  following  sections. 

1.   Inadequate  and  uncoordinated  planning  efforts  early  in  the 

development  cycle  and  among  the  various  responsible  organiza- 
tions 

Inadequate  T&E  planning  has  been  the  source  of  many  difficulties, 
and  has  resulted  in  some  cost  and  schedule  growth  in  development  programs. 
Planning  simply  was  not  being  performed  early  enough  in  the  development 
cycle,  and  when  it  was  performed,  many  times  it  was  incomplete,  cover- 
ing primarily  DT&E.   This  demonstrated  the  lack  of  the  user's  input. 
As  a  result  initial  program  budget  submittals  at  Milestone  I  were  very- 
deficient  in  the  T&E  area. 

The  requirement  for  a  TEMP  to  exist  at  the  major  milestones 
(including  Milestone  I)  for  each  program  is  an  attempt  to  correct  this 
problem.   Unfortunately,  many  TEMPs  do  not  exist  until  Milestone  II, 
thus  allowing  the  opportunity  for  many  differences  between  what  is  pre- 
sented at  Milestone  I  and  Milestone  II.   In  discussions  with  OPNAV  983, 
it  was  brought  out  that  only  a  small  percentage  of  program  TEMPs  are 
approved  by  Milestone  II  (a  number  as  low  as  10%) .   This  may  be  because 
the  requirement  for  a  TEMP  is  relatively  new  (since  1975) ;  however,  it 
may  also  be  an  example  of  people  dragging  their  feet  on  implementing 
directives.   There  appears  to  be  no  sanction  of  the  document  and  no 
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penalty   against  programs  which  do  not  have  a  TEMP.   Also,  NAVAIR  has 
yet  to  flow  down  implementation  and  guideline  instructions  to  the  upper 
level  directives. 

Many  times  the  T&E  program  plans  submitted  were  not  coordinated 
between  the  various  T&E  organizations  and  responsible  program  people. 
This  situation  may  exist  when  a  program  does  not  have  a  T&E  coordinator 
or  the  Program  Manager  allows  disjointed  efforts  to  exist.   Adequate 
early  planning  would  minimize  this  situation.   Again,  the  requirement 
for  a  TEMP  is  an  attempt  to  obtain  a  coordinated  T&E  program.   A  coordi- 
nated program  can  also  help  to  reduce  another  major  problem,  redundant 
testing. 

The  Defense  Science  Board,  in  the  reports  of  its  task  force  on 
T&E,  reported  an  indication  of  widespread  inadequate  early  planning  for 
T&E  in  past  programs  [Refs.  14  and  15]. 

2.  Implementation  of  "success-oriented"  planning 

When  test  planning  is  performed,  it  generally  includes  all  known 
testing  requirements  and  contains  some  integration  of  the  testing  activ- 
ities.  However,  the  planning  is  basically  "success-oriented",  whereby 
test  failures  with  corresponding  contingencies  are  not  included  in  the 
plans.   Only  in  flight  testing  are  back-up  tests  consistently  identified, 
although  many  times  insufficient.   Ground  testing,  especially  in  DT&E, 
suffers  from  this  characteristic.   The  Defense  Science  Board  in  its  most 
recent  review  on  T&E  also  stated  that  success-oriented  planning  was 
performed  [Ref .  15] . 

3.  Lack  of  flexibility  in  T&E  planning  to  accommodate  problems  and 
unknowns 

Many  times  the  test  program  planned  did  not  allow  sufficiently 

for  problem  solving  and  unknown  situations  which  surface  during 
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development,  contributing  to  schedule  stretchout  and  increased  cost  in 
test  programs.   This  is  a  very  difficult  feature  to  plan  for,  but  one 
to  which  the  Program  Manager  should  give  more  attention  in  future  plan- 
ning.  Many  times  problems  are  uncovered  during  testing,  or  some 
unknown  occurs,  which  necessitate  changes  in  the  course  of  action.   How- 
ever, a  real  difficulty  exists  in  that  many  times  program  funds  and 
schedules  do  not  possess  the  flexibility  required  to  make  the  necessary 
program  adjustments.   This  situation  was  highlighted  by  the  Defense 
Science  Board  [Refs.  14  and  15].   In  discussions  they  had  with  industry 
representatives,  industry  indicated  almost  universally  an  erosion  process 
of  program  contingency  funds  throughout  the  bidding  and  negotiation 
process. 

4.  Unrealistic  requirements  in  the  planning  effort 

Since  the  implementation  of  DoDD  5000.3  in  1973,  improved  plan- 
ning has  taken  place  in  T&E.   The  OT&E  people  of  CCMOPTEVFOR  were  given 
a  strengthened  charter  which  they  have  proceeded  to  implement.   However, 
the  organizations  responsible  for  DT&E  have  existing  "standard  operating 
procedures"  which  they  implement.   Combining  these  two  perceptions  into 
the  test  program  plans  resulted  in  an  abundance  of  testing  being  speci- 
fied, with  high  cost  T&E  programs.   During  this  period,  studies  were 
reporting  that  inadequate  and  insufficient  T&E  was  being  performed. 
Future  planning  must  look  for  most  cost-effective  T&E  programs . 

5.  Lack  of  design  of  tests,  definition  of  test  criteria,  and  know- 
ing the  use  cf  the  test  data 

Poor  planning  in  the  design  of  tests,  definition  of  test 

criteria,  and  in  not  knowing  the  expected  use  of  the  test  data  results 

in  proliferation  of  testing,  as  well  as  wasteful  testing  and  testing 

with  questionable  results.   Such  a  situation  should  not  exist  in  today's 
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United  funding  environment.   Testing,  especially  flight  testing,  is 
very  expensive.   This  problem  includes  both  DT&E  and  OT&E.   The  OT&E 
people  must  be  especially  concerned  with  this  situation  because  of 
their  desire  to  test  in  the  operational  environment  which  is  more 
difficult  and  costly. 

The  Defense  Science  Board  report  contains  several  guidelines 
relative  to  design  of  tests  which  are  appropriate  to  consider.  They 
include: 

a.  "Ensure  that  the  whole  system,  including  the  user  people, 
is  tested.   Realistically  test  the  complete  system,  in- 
cluding hardware,  software,  people  and  interfaces. 

b.  Ascertain  that  sufficient  time  and  test  articles  are 
planned.   When  the  technology  is  stressed,  the  higher 
risks  require  more  test  articles  and  time. 

c.  In  general,  parts,  sub-systems  and  systems  should  be 
proven  in  that  order  before  incorporating  them  into  the 
next  higher  assembly  for  more  complete  tests. 

d.  Major  tests  should  never  be  repeated  without  an  analysis 
of  failures  and  corrective  action"  [Ref .  7] . 

6.   Lack  of  definition  of  a  mission  profile 

In  recent  years,  the  value  of  a  mission  profile  or  the  several 
mission  profiles  of  a  weapon  system  has  been  realized  and  become  a  re- 
quirement for  programs.   This  resulted  from  a  weakness  in  the  user-pro- 
ducer dialogue.   The  operational  requirements  of  the  user  were  not  get- 
ting into  the  development  specifications.   The  DSB  refers  to  this 
recommending  that  specification  requirements  be  stated  in  functional 
terms  rather  than  design  values,  necessitating  strong  user  involvement 
[Ref.  14]  . 

Specifying  a  system's  mission  profile  requires  knowledge  of  the 
operational  performance  requirements,  and  incorporates  a  realism  into 
requirements  and  subsequent  testing.   Obtaining  a  system's  mission 
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profile  requires  planned  and  coordinated  efforts  by  the  program's  using, 
development,  and  testing  teams.   The  lack  of  this  in  the  past  has  con- 
tributed to  wasteful,  inefficient  and  less  than  desirable  T&E  efforts. 

7.   Difficulty  in  converting  operational  requirements  into  test 
specification  requirements 

This  has  been  a  problem  for  many  years  and  probably  will  continue 
to  be  a  prime  area  for  improvement  in  the  future.   Better  early  planning 
can  minimize  this  difficulty,  along  with  an  improved  user-producer  dia- 
logue.  During  the  generation  of  the  system's  performance  and  design 
specifications,  operational  requirements  should  be  the  source  for  these 
specifications,  with  strong  user  involvement.   At  the  same  time,  the  user- 
producer  dialogue  must  be  extended  to  include  test  program  planning  with 
the  specification  generation  task.   Only  if  this  is  done  will  the  DT&E 
and  OT&E  programs  be  testing  real  operational  or  operational  related  re- 
quirements.  The  Defense  Science  Board  in  its  most  recent  study  cited 
inadequate  requirements  statements  as  a  problem  [Ref .  15] . 

Currently  OPTEVFOR  is  closest  to  the  operational  community; 
however,  in  many  instances  its  personnel  lack  expertise  to  translate  the 
perceived  operational  requirements  into  development  specification 
language.   Also,  OPUAV,  which  is  a  user  organization,  appears  to  be  often 
influenced  by  technical  inputs  from  contractors  and  development  labora- 
tories.  Thus,  the  operational  requirements  may  be  losing  user  represen- 
tation. 

3.   Inability  to  convert  specification  performance  requirements  into 
T&5  asset  (resource)  requirements 

When  the  test  agencies  have  made  their  inputs,  and  the  test  per- 
formance requirements  have  been  established,  the  problem  exists  of  con- 
verting these  into  resource  requirements.   It  is  important  to  perform  a 
requirements  analysis  in  conjunction  with  the  design  of  the  test  to 
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determine  the  asset  limitations,  both  from  feasibility  and  affordability 
points  of  view.   The  testing  required  to  perform  the  desired  program  is 
simply  resource  limited.   Among  resource  considerations  are  such  items 
as  facilities,  personnel,  test  hardware,  threat  simulation,  and  support 
equipment.   Both  the  NMARC  study  [Ref.  16]  and  Defense  Science  Board  re- 
port [Ref.  14]  identified  this  limitation  in  Navy  T&E.   Test  resources 
are  not  keeping  pace  with  development  program  requirements. 

In  general,  the  Navy  must  continue  to  make  improvements  in  its 
T&E  planning.   With  progress  being  made  through  the  implementation  of 
DoDD  5000.3  and  OPNAVINST  3960.10,  and  the  subsequent  TEMP,  plans  for 
T&E  should  be  improved.   However,  this  result  is  not  yet  clearly  apparent. 
Much  work  is  being  expended,  supported  by  the  fact  that  many  TEMPs  are 
in  progress  or  draft  form,  but  only  a  small  percentage  are  formally 
approved.   Additional  efforts  to  perform  planning  earlier  in  the  devel- 
opment phases  would  prove  to  be  beneficial.   In  addition  to  providing 
the  Program  Manager  with  valuable  management  information,  such  early 
planning  could  be  used  as  a  "strawman"  for  the  TEMP,  thus  possibly  re- 
ducing its  generation  and  approval  cycle  time.   This  early  planning 
would  also  highlight  any  potential  T&E  gaps  which  impact  on  program 
milestone  decisions  (T&E  gaps  are  breaks  in  testing  due  to  hardware  un- 
availability, anywhere  during  the  development  and  production  cycle,  not 
just  between  IOT&E  and  FOT&E  as  used  in  Ref.  14) .   This  early  planning 
must  also  involve  very  strongly  the  user  and  OPTEVFOR. 

B.   PROLIFERATION  OF  REQUIREMENTS  AND  REVIEWS 

During  the  performance  of  T&E  throughout  the  development  cycle,  new 
requirements  surface  and  are  imposed  on  existing  programs  as  well  as 
new  ones.   These  requirements  are  not  planned  for  because  they  did  not 
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exist  at  the  time  of  planning.   Such  requirements  might  be  in  the  form 
of  AR's  (Aeronautical  Requirements)  which  are  essentially  new  require- 
ments which  generally  affect  testing  procedures.   Other  requirements 
might  be  program  reviews  which  were  not  anticipated  and  may  reflect  a 
current  fleet  concern  or  a  shift  in  emphasis  in  certain  disciplines 
(e.g.,  reliability).   Examples  of  such  reviews  at  the  Washington  level 
were  the  developing  agency  special  management  reviews  addressing  heavily 
the  reliability  concern,  and  the  production  reliability  design  review 
[Refs.  17  and  18] .   Progress  on  the  development  program  and  in  T&E  is 
affected  by  such  reviews  because  of  the  support  required  to  prepare  and 
perform  the  reviews.   Reviews  such  as  these  are  not  provided  for  in 
program  budgets  and  contribute  to  a  depletion  of  the  management  discre- 
tionary funds.   Review  requirements  such  as  these  also  seem  to  come  and 
go,  depending  on  the  people  in  the  review  chair  or  those  in  power.   The 
KI1ARC  report  cited  the  difficulty  the  Program  Manager  has  responding  to 
all  reviewers  and  requirements  from  the  many  layers  of  people  in  the 
Navy  organization  [Ref .  16] . 

There  are  other  reviews  which  impact  on  T&E  as  well.   Reviews  at 
participating  field  activities  such  as  design  reviews,  environmental  and 
qualification  program  reviews,  and  those  related  to  range  safety  require 
special  efforts  from  the  T&E  organization  and  many  times  are  beyond 
that  planned  in  the  budget.   Proper  attention  to  such  requirements  in 
early  planning  could  minimize  this  impact.   However,  a  proliferation  of 
these  requirements  may  be  detrimental  to  efficient  development  progress. 

Unique  capabilities  may  fall  into  the  category  of  proliferation. 
For  example,  some  test  activities  may  have  built  up  capabilities  over 
the  years,  both  in  facilities  and  manpower,  which  are  quite  special, 
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and  these  activities  tend  to  perpetuate  the  use  of  such  capabilities 
and  their  existence.   Capabilities  such  as  PMTC,  Point  Ilugu's  acoustic 
vibration  facility  and  Naval  Explosive  Ordnance  Disposal  Facility  (NEODF) 
Indian  Head's  propellant  manufacturing  and  testing  capability  fall  into 
this  category.   Although  these  facilities  and  capabilities  are  important 
ones,  they  may  not  fit  into  every  program's  T&E  plans.   However,  once 
activities  possess  a  unique  capability,  the  activities  are  determined 
to  keep  it  in  use.   Thus,  requirements  may  be  imposed  on  the  program  to 
include  the  use  of  these  special  test  facilities  as  part  of  the  test  pro- 
gram.  The  more  logical  approach  is  for  the  program  to  determine  what 
testing  is  required  and  then  determine  the  best  method  and  place  to 
accomplish  that  testing. 

C.   RESOURCE  AND  SCHEDULE  CONSTRAINTS 

Resource  and  schedule  constraints  add  to  the  complexity  of  accomplish- 
ing the  T&E  assignment.   These  constraints  have  been  used  as  an  excuse 
for  non-compliance  with  directives  and  instructions  applicable  to  defense 
development  programs  [Ref .  14] . 

The  primary  resources  for  T&E  are  personnel,  facilities  and  equipment. 
Talented  T&E  people  are  key  to  performing  the  T&E  function.   T&E  suffers 
from  a  lack  of  talented  people  and  this  area  must  be  made  more  attractive 
and  rewarding.   Facilities,  ranges,  instrumentation  and  support  equip- 
ment are  growing  obsolete  and  have  maintenance  problems.   With  the  de- 
crease of  defense  funding  in  the  1970 's,  less  flight  testing  is  performed 
and  fewer  dollars  are  available  to  maintain  ranges  and  facilities  at 
acceptable  levels.   In  addition,  there  is  a  need  to  incorporate  improve- 
ments necessary  to  keep  pace  with  the  newer,  more  sophisticated  missile 
systems  currently  being  developed.   Support  equipment  is  often  overlooked 
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in  the  choice  for  new  targets,  range  upgrading  or  repair,  but  equipment 
such  as  range  trackers  and  telemetry  systems  are  equally  important  in 
achieving  satisfactory  T&E.   The  NMARC  report,  the  Defense  Science  Board 
reports  and  discussions  with  the  NWC,  China  Lake  T&E  facility  manager 
all  highlighted  this  problem  relating  to  T&E  resources  [Refs.  14,  15, 
and  16] . 

A  recent  concern  related  to  maintaining  adequate  test  ranges  has 
been  encroachment  by  civil  activities  on  the  government's  vast  range 
land  and  the  sea  and  airspace  needed  for  testing.   Such  a  movement  is 
a  potential  threat  to  this  test  resource.   Lt  Gen  W.  E.  Lotz  (Ret) , 
head  of  DD(T&E) ,  cited  this  problem  as  very  real  in  his  presentation  to 
Congress  on  defense  T&E  [Ref .  3] . 

Schedule  constraints  may  be  the  single  greatest  excuse  given  for 
not  accomplishing  the  T&E  plans  or  why  T5E  is  performed  out  of  sequence 
with  program  milestones .   Usually  programs  are  striving  to  meet  or  better 
a  schedule,  but  many  times  there  is  some  program  slippage.   When  program 
decision  points  are  based  on  a  predetermined  schedule,  many  times  the 
decision  milestone  takes  place  and  the  following  phase  is  initiated 
before  the  T&E  is  completed,  because  T&E  characteristically  is  more 
prevalent  in  the  latter  part  of  the  development  phase.   The  report  of 
the  Commission  on  Government  Procurement  cited  the  existence  of  this 
situation  and  recommended  instead  that  milestone  decisions  be  based  on 
program  information  and  T&E  results  [Ref.  11] .   The  Defense  Science 
Board  also  noted  this  situation  and  made  a  similar  comment  [Ref.  15] . 
However,  schedule  constraints  continue  to  prevail  because  people  in 
command  are  uncomfortable  with  flexible  milestones  and  the  budget  cycle 
operates  from  a  fixed  schedule  base. 
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DD(T&E)  does  make  a  najor  recommendation  to  DDR&E  at  the  program 
milestone  reviews.   However,  rarely  has  a  program  been  stopped  because 
of  insufficient  T&E.   This  occurs  because  there  are  very  few  yes  and 
no  answers  in  T&E,  creating  a  wide  zone  of  uncertainty,  risk  and  judge- 
ment evaluation.   But,  as  the  Defense  Science  Board  reported,  DD(TSE) 
and  the  milestone  decision  makers  have  been  unable  to  make  risk  assess- 
ments, and  the  program  has  moved  on  according  to  the  calendar  driven 
schedule  [Ref .  15] .   Management  must  better  address  this  situation, 
develop  new  plans  and  procedures,  and  then  adhere  to  performance  driven 
milestones.   The  latest  directives  relative  to  system  acquisition, 
currently  in  review,  are  attempting  to  mandate  performance  driven  mile- 
stones rather  than  schedule  driven  milestones. 

D.   REDUNDANT  TESTING 

Redundant  testing  is  testing  more  than  is  necessary  through  duplica- 
tive efforts  among  various  test  agencies.   The  potential  for  increased 
redundant  testing  has  occurred  since  the  issuance  of  the  new  directives 
in  the  1970s  with  their  added  emphasis  on  test  and  evaluation.   These 
new  directives,  as  applied  to  the  Navy,  gave  OPTEVFOR  additional  respon- 
sibility and  stature,  resulting  in  an  expansion  of  OT&E  testing  require- 
ments.  The  requirements  of  DTSE  have  remained  the  same  with  some 
inherent  redundant  testing  built  into  the  development  program  as  the 
government  provided  a  "watchdog"  function  on  the  contractor.   The  com- 
bined effect  of  the  OT&E  and  DT&E  requirements  results  in  additional 
duplication  of  tests.   Prime  examples  of  this  situation  in  ALGM  programs 
are  in  environmental  testing,  reliability  testing,  and  captive  flight 
testing. 
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In  environmental  testing  the  development  contractor  is  required  by 
his  contract  to  perform  environmental  tests  on  the  system  and  applicable 
sub-systems.   The  llavy  then  proceeds  to  repeat  much  of  the  testing  dur- 
ing its  TECHEVAL  at  PMTC,  Point  Mugu.   This  is  classified  as  confirma- 
tion of  the  contractor's  tests.   In  the  past,  much  of  this  type  of  test- 
ing was  performed  to  prepare  or  train  Point  Mugu  for  its  role  in  fleet 
related  testing  after  IOC.   Such  redundant  testing  simply  depletes  the 
already  strained  RDT&E  budget. 

The  situation  in  reliability  testing  could  become  quite  serious. 
There  has  been  much  emphasis  on  reliability  in  the  past  several  years 
because  of  high  maintenance  costs  and  fleet  disappointment  in  weapon 
systems  that  did  not  perform  at  all  or  to  their  advertised  capability. 
As  a  result,  many  T&E  organizations  are  proposing  their  own  reliability 
test  and  demonstration  program  for  the  same  weapon  system.   Also,  many 
people  in  program  review  positions  are  encouraging  and/or  requiring  the 
demonstration  test  programs.   However,  the  fact  must  not  be  overlooked 
that  such  testing  is  quite  expensive  and  a  large  amount  of  testing  is 
required  for  any  one  organization  to  be  able  to  state  a  reliability 
from  a  demonstration  program  with  any  high  level  of  confidence.   For 
example,  if  a  missile  has  a  reliability  requirement  of  75  hrs  MTBF,  700 
hrs  of  operating  time  with  no  more  than  5  failures  are  required  to 
demonstrate  at  the  90%  confidence  level  the  specified  reliability  require- 
ment (based  on  the  exponential  distribution  assumption) .   The  contractor 
is  contractually  required  to  demonstrate  the  reliability  of  the  system. 
That  alone  requires  a  substantial  amount  of  testing.   A  requirement  for 
TECHEVAL  people  and  the  OT&E  people  also  to  perform  reliability  demonstra- 
tion test  programs  is  often  uneconomical  and  redundant.   Each 
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organization  would  claim,  though,  that  its  demonstration  program  was 
different  because  of  environmental  changes. 

In  captive  flight  testing  of  ALGMs,  tests  are  performed  by  the  de- 
velopment agency  (contractor  and  Navy  development  laboratory)  as  part 
of  DT&E,  the  TECHEVAL  agency,  also  as  part  of  DT&E,  and  OPTEVFOR  as 
part  of  OT&E.   Although  some  unique  tests  are  performed  by  each  test 
agency,  there  is  some  duplication  as  well.   Claims  of  program  advocacy 
by  the  development  agency  and  testing  by  "independent"  test  agencies 
are  reasons  used  to  justify  such  redundant  testing. 

The  question  of  redundant  testing  must  be  examined  carefully  before 
implementation  because  of  the  tight  fiscal  situation  of  present  times. 
It  appears  reasonable  that  through  proper  planning  of  the  development 
contractor's  test  program  and  associated  monitoring  by  the  Navy,  some 
redundant  testing  could  be  eliminated.   In  addressing  the  problem  of 
redundant  testing,  the  objective  should  not  be  to  remove  all  duplica- 
tion.  It  should  ensure  that  where  such  duplication  exists,  it  is 
visible,  affordable,  controlled,  purposeful,  and  contributes  appro- 
priately to  the  T&E  program.   The  Defense  Science  Board  noted  that  there 
appeared  to  be  little  or  no  overtesting  done  under  the  current  direc- 
tives, but  seme  redundant  testing  occurred  [Ref.  15]. 

E.   MANAGEMENT  OF  DIVERSE  ACTIVITIES  AND  FUNCTIONS 

One  of  the  significant  difficulties  which  is  often  overlooked  in 
managing  T&E  is  that  the  Program  Manager  and  T&E  coordinator  must  deal 
with  many  diverse  activities  and  functions.   In  addition  to  DD(T&E)  in 
OSD,  OPTEVFOR  and  OP-983  in  OPNAV,  the  T&E  manager  must  operate  with 
the  matrix  organizational  structure  of  the  Navy  for  his  functional  T&E 
and  technical  support.   Many  of  the  T&E  activities  and  organizations 
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within  the  Navy  possess  their  own  standard  operating  procedures  and  ex- 
hibit various  forms  of  parochialism.  Dealing  with  this  characteristic 
requires  special  talents  and  capabilities  by  the  T&E  manager  in  order  to 
operate  effectively.   The  important  point  is  to  realize  everyone's  posi- 
tion when  the  T&E  planning  takes  place. 

Management  of  the  Navy  and  development  contractor  activities  simul- 
taneously, their  interfaces  and  interfacing,  is  another  formidable  task 
for  the  ALGM  Program  Manager.   Many  times  competition  and  adversary  rela- 
tionships exist  between  the  two  groups.   In  T&E  it  is  often  the  Navy's 
position  that  the  contractor  has  something  to  gain  by  taking  advantage 
of  the  Navy  and  will  do  so  in  its  testing  program.   There  is  disagree- 
ment between  the  two  organizations  performing  the  tests  partly  because 
the  contractor  feels  it's  his  responsibility  and  the  Navy  feels  it  must 
perform  the  tests  to  validate  the  results.   Much  of  the  current  thinking 
in  ALGM  programs  is  that  the  system  development  contractor  is  responsible 
for  all  the  development  activities,  including  the  many  aspects  of  T&E 
from  component  and  assembly  testing  to  system  flight  testing.   This 
creates  a  management  problem  in  that  the  Navy  is  responsible  for  the 
aircraft  and  test  ranges  where  the  system  flight  testing  is  performed. 

In  summary,  it  is  necessary  for  the  ALGM  Program  Manager  to  be  con- 
scious of  each  problem  in  order  for  him  to  better  manage  and  implement 
a  successful  and  effective  T&E  effort.   Sometimes  the  problem  situation 
is  of  such  a  nature  that  the  ALGM  Program  Manager  has  little  opportunity 
to  exert  influence  on  or  correct  the  ultimate  outcome;  however,  simply 
realizing  the  problem  exists  helps  him  tolerate  the  situation.   Never- 
theless, many  of  the  T&E  management  problems  can  be  dealt  with  directly 
by  the  ALGM  Program  Manager. 
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The  T&E  management  problems  identified  in  this  section  will  be 
addressed  in  the  following  discussion  of  potential  alternatives  for 
performing  an  ALGM  program's  T&E.   Some  of  the  problems  will  be  more 
prevalent  in  one  alternative  than  in  another. 
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V.   AIR-LAUNCHED  GUIDED  MISSILE  T&E  PROGRAM 
MANAGEMENT  ALTERNATIVE  APPROACHES 

The  Program  Manager  has  several  basic  approaches  or  alternatives 
with  respect  to  implementation  of  an  "adequate"  T&E  Program  for  an  ALGM. 
Four  alternatives  will  be  considered  in  this  section,  all  of  which  are 
basically  compatible  with  the  requirements  and  guidelines  of  DoDD  5000.3 
and  OPIIAVINST  3960.10.   The  four  alternatives  are: 

A.  Separate  Test  Programs  in  Series 

B.  Combined  Test  Programs 

C.  Integrated  Test  Programs 

D.  System  Level  Functional  Test  Program 

Each  alternative  when  analyzed  suffers  to  a  certain  extent  from  the  T&E 
management  problems  identified  in  section  IV.   The  applicability  of 
those  management  problems  and  other  problems  to  a  particular  alternative 
is  discussed  in  this  section.   The  good  features  of  each  approach  are 
also  discussed. 

A.   SEPARATE  TEST  PROGRAMS  IN  SERIES 
1.   Management  Approach 

The  Separate  Test  Programs  in  Series  management  approach  adheres 
rigidly  to  the  requirements  and  guidelines  of  OPNAVINST  3960.10  and  is 
a  means  of  implementing  DT&E  and  OT&E  efforts  similar  to  those  described 
in  section  IIIc.   The  Program  Manager  attempts  to  structure  the  T&E 
program  utilizing  contractors,  participating  field  activities,  and 
support  activities.   Each  program  T&E  participant  will  advocate  and 
attempt  to  conduct  its  own  test  program  to  demonstrate  ALGM  specification 
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and  or  DCP  compliance.   The  nature  and  degree  of  independence  of  each 
participant  will  be  reflected  in  the  structured  inputs  to  the  TEMP 
preparation  and  review  cycle  resulting  in  basically  three  separate 
test  programs  being  conducted  as  follows: 

a.  Contractor  Program 

DT-I  through  DT-III  as  described  in  section  III  C 1  culmi- 
nating with  Contractor  Demonstration  Tests,  including: 

(1)  Part  and  component  qualification,  assembly,  and  sub- 
system tests  (accelerated  life,  overstress,  and  design 
evaluation  and  qualification) ,  sub-system  and  section 
integration  and  compatibility  tests. 

(2)  ALGM  system  level  performance  tests  both  on  the  ground 

(chamber  tests) ,  on  board  the  aircraft,  and  in  the 
captive  carry  mode  concluding  with  several  missile 
launches  from  aircraft, 

(3)  reliability  demonstration  for  compatibility  with 
specification  requirements  (ground  and  airborne) 
testing  for  captive  carry  MTBF,  launch  from  aircraft 
and  free  flight  to  target  reliability)  at  the  required 
confidence  levels, 

(4)  maintainability  demonstration  for  compatibility  with 
specification  requirements, 

(5)  environmental  and  safety  qualification  (including 
electromagnetic  compatibility)  to  verify  compliance 
with  specification  design  requirements. 

b.  TECHEVAL  Program 

Performed  during   the  final   sub-phase  of  DT-III   and 
includes : 
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(1)  ALGM  system  level  performance  both  on  the  ground 

(chamber  tests) ,  on  board  the  aircraft,  and  in  the 
captive  carry  mode  concluding  with  several  missile 
launches  from  aircraft, 

(2)  reliability  demonstration  for  compatibility  with 
specification  requirements  (ground  and  airborne  testing 
for  captive  carry  MTBF,  launch  and  free  flight  to 
target  reliability)  at  the  specification  required 
confidence  levels, 

(3)  maintainability  evaluation, 

(4)  environmental  tests  to  verify  compliance  with  speci- 
fication design  requirements. 

Occasionally,  the  TECHEVAL  environmental  tests  have  included  overstress 
tests  to  evaluate  hardware  design  margins, 
c.   OT&E  Program 

OT-II  and  OT-III  as  described  in  section  III  C  2  culminating 
with  OPEVAL,  which  includes: 

(1)  ALGM  system  level  performance  tests  on  board  the  air- 
craft, on  the  ground  and  in  the  captive  carry  mode 
concluding  with  several  missile  launches  from  the 
aircraft, 

(2)  airborne  evaluation  of  captive  carry  MTBF  and  free 
flight  to  target  reliability  for  compliance  with  the 
ALGM  system  specification  confidence  levels, 

(3)  evaluation  of  ALGM  system  maintainability 
characteristics . 

Other  operational  suitability  characteristics  of  the  ALGM  evaluated 
are:   compatibility,  interoperability  including  human  factors,  and 
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logistic  and  training  adequacy.   Additionally,  OPTEVFOR  will  continue 
tactics  development  and  refinement. 

d.   FOT&E  Program 

OT-IV,  as  described  in  section  IIIC3,  may  be  conducted  by 
OPTEVFOR  but  is  rarely  defined  and/or  planned  for  early  in  the  program's 
life.   FOT&E  is  beyond  the  scope  of  this  thesis. 
2.   Applicable  T&E  Management  Problems 

The  principal  Developing  Agency  for  Navy  ALGM's  is  the  Naval  Air 
Systems  Command  and  AIR-06  is  its  responsible  T&E  agency  (Figure  6)  . 
AIR-06  has  been  in  existence  for  approximately  three  years  and  has  not 
as  yet  promulgated  T&E  Instructions  or  Guidelines  in  response  to 
OPNAVINST  3960.10.   This  has  contributed  to  something  less  than  full 
implementation  of  DoDD  5000.3  and  OPNAVINST  3960.10  T&E  requirements 
and  permits  circumstances  such  as  the  Separate  Test  Programs  in  series 
to  occur  too  easily  rather  than  explore  a  more  efficient  process. 
According  to  AIR-06  personnel,  it  has  been  very  difficult  for  AIR-06  to 
effectively  impact  ALGM  T&E  programs  and  to  fully  and  efficiently  im- 
plement the  TEMP  process  because  of  the  lack  of  published  T&E  instruc- 
tions.  A  further  constraint  to  date  on  AIR-06 's  effectiveness  is  the 
fact  that  its  personnel  have  been  utilized  for  the  most  part  only  in  an 
advisory  capacity  by  ALGM  Program  Managers.   Thus,  without  more  specific 
guidance  and  direction  from  within  NAVAIR,  TEMP  preparation  has  been 
dragging  and  ALGM  T&E  management  problems  similar  to  those  described  in 
section  IV  continue.   T&E  management  problems  arising  from  the  Separate 
Test  Programs  in  Series  method  of  operation  are  as  follows: 

a.   Planning 

As  previously  indicated  in  section  IVA.l,  NAVAIR1 s  current 
position  with  respect  to  flowdown  of  T&E  implementation  and  guideline 
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instructions,  makes  it  difficult  to  influence  or  even  impress  upon  ALGM 
Program  Managers  the  necessity  for  early,  thorough  T&E  planning.   Further 
complicating  this  situation,  is  the  fact  that  DD(T&E),  OP-983C,  and  NAVMAT 
have  all  indicated,  when  interviewed  by  the  authors,  that  a  strawman  TEMP 
is  all  that  is  required  and  reasonable  to  expect  for  Milestone  I.   It  was 
stated  also  that  OPTEVFOR  was  usually  not  available  for  early  planning 
efforts  prior  to  the  Milestone  I  TEMP  effort  because  of  its  heavy  involve- 
ment in  OPEVALs.   A  contradiction  exists  in  that  OPTEVFOR  personnel,  when 
interviewed,  indicated  problems  with  early  planning  prior  to  Milestone  I 
were  due  to  reluctance  on  the  part  of  Program  Managers  to  commit  to 
supporting  a  detailed  T&E  program.   OPTEVFOR  further  stated  that  there 
were  no  real  sanctions  imposed  on  the  Program  Managers  or  their  programs 
if  they  did  not  do  the  necessary  thorough  investigation  and  planning 
effort.   Too  often,  Program  Managers  are  unable  or  unwilling,  because 
of  the  myriad  of  tasks  in  the  early  stages  of  the  program,  to  come  to 
grips  with  the  T&E  details  and  to  challenge  or  even  question  established 
precedents  or  standard  operating  procedures.   Under  these  circumstances, 
T&E  programs  like  the  Separate  Test  Programs  in  series  can  result.   Once 
brought  to  life  at  Milestone  I,  the  drain  on  scarce  program  resources 
commences  and  is  extremely  difficult  to  curb  and  control, 
b.   Proliferation  of  Requirements 

With  the  Separate  Test  Programs  in  series  approach  the  Pro- 
gram Manager  is  much  more  likely  to  be  confronted  with  proliferating 
requirements  which  impinge  on  scarce  program  resources .   ACAT  I  or  II 
programs  usually  have  many  diverse  participants  (participating  field 
activities,  such  as  :  PMTC,  NWC,  NWL)  who  have  unique  internal  require- 
ments which  must  be  satisfied  during  the  life  of  the  program.   These  re- 
quirements may  take  the  form  of  management  review,  special  technical 
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reviews,  special  test  programs  or  regimes,  and  additional  requirements 
designed  to  perpetuate  utilization  of  unique  facilities  or  capabilities. 
For  example,  NWC,  which  has  functioned  as  Technical  Manager  for  NAVAIR 
on  several  ALGM  programs  and  as  a  participating  field  activity  develop- 
ing major  components  on  other  ALGM  programs,  has  internal  instructions 
which  require  extensive  management  reviews  (e.g.,  Design  Review  Committee, 
Environmental  Qualification  Review  Panel,  Ammunition  Safety  Committee) . 
Any  or  all  of  these  retain  the  authority  to  impose  requirements  for 
additional  program  resources,  i.e.,  additional  ALGM  assets  for  testing 
(larger  quantities) ,  additional  tests  or  sequences  which  may  require 
different  or  enhanced  facilities  or  more  schedule  time. 

TECHEVAL's  conducted  at  PMTC  invariably  "require"  subjection 
of  ALGM's  to  extensive  environmental  tests  not  only  to  verify  compliance 
with  specification  design  requirements  but  also  to  evaluate  design 
margins  (i.e.,  overstress  or  test  to  failure  type  of  tests).  PMTC  also 
is  a  proponent  of  acoustic  vibration  chamber  testing  and  possesses  one 
of  the  few  facilities  of  this  type  capable  of  testing  all-up-round  ALGM's. 
Consequently,  ALGM  ground  reliability  tests  (simulating  the  captive 
flight  environment)  are  conducted  to  demonstrate  the  specified  MTBF  at 
the  required  confidence  level  using  this  chamber.   This  is  required 
even  though  the  ALGM  contractor  has  already  performed  a  similar  demonstra- 
tion to  satisfy  contractual  provisions  under  a  government  approved  pro- 
cedure with  government  monitoring. 

The  Program  Managers  are  often  faced  with  high  level  manage- 
ment review  similar  to  those  reported  by  NMARC  [Ref .  16] .   The  reviews 
utilize  T&E  data  and  are  usually  conducted  at  such  a  high  level  (Flag 
Rank)  that  the  program  essentially  comes  to  a  halt  while  presentations 
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are  prepared,  participants  assemble,  and  the  review  occurs.   Quite  com- 
monly, this  occurs  several  times  during  ALGM  development  and  impacts 
significantly  on  program  resources. 

With  the  increased  emphasis  on  IOT&E,  OPTEVFOR  is  involved 
in  monitoring  DT&E  activities.   In  addition  to  operational  testing, 
OPTEVFOR  personnel  have  said  in  recent  interviews  that  they  may  choose 
to  repeat  DT&E  testing  where  apparent  weaknesses  were  revealed, 
c.   Resource  and  Schedule  Constraints 

The  Separate  Test  Programs  in  Series  approach  can  present 
especially  significant  resource  and  schedule  constraint  problems  for 
the  Program  Manager  or  his  designee  for  T&E.   Inadequate  early  coordi- 
nated planning  impacts  severely  on  lead  times  necessary  to  acquire 
requisite  facilities  and  instrumentation  with  appropriate  staffs  which 
may  be  necessary  to  support  new  ALGM  technology.   A  further  problem 
aggravating  this  situation  is  the  fact  that  the  Navy's  Test  and  Evalua- 
tion Facility  base  is  still  loosely  knit  and  without  strong  central 
coordination.   NMARC  [Ref.  16]  commented  on  this  problem  and  made 
several  recommendations  which  should  improve  the  situation  as  NAVAIRs 
relatively  new  T&E  Directorate,  (AIR-06) ,  assumes  its  leadership  role. 

With  the  high  possibility  for  duplicative  testing  under 
the  Separate  Test  Programs  in  Series  approach,  problems  like  facility 
scheduling  and  allocation  of  priorities,  become  quite  significant. 
OPTEVFOR  possesses  no  facilities,  instrumentation,  or  ranges  of  its  own 
and,  because  of  the  nature  of  some  ALGM's,  must  rely  on  DT&E  resources 
to  conduct  tests  for  it.   Certain  range  and  instrumentation  capabilities 
available  only  at  government  facilities  are  required  throughout  the 
ALGM  development  cycle  and  present  a  constant  scheduling  problem  because 
of  contractor,  Navy  lab,  and  OPTEVFOR  requirements. 
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Another  aspect  of  the  resource  and  schedule  constraints 
related  to  facilities,  instrumentation,  and  ranges  is  the  impact  of 
obsolescence  and  increased  maintenance  costs.   The  Navy's  facilities 
are  having  a  difficult  time  keeping  pace  with  technological  advances 
in  weaponry  and  expanded  emphasis  on  testing  is  requiring  more  main- 
tenance on  existing  facilities  and  instrumentation.   This  was  confirmed 
in  a  recent  interview  with  the  Head  of  the  NWC  T&E  organization,  where 
many  ALGM  t'ests  are  conducted. 

d.  Redundant  Testing 

Contractor  contractually  required  Reliability  Demonstration 
(both  ground  and  airborne)  and  Environmental  Qualficiation  tests  are 
repeated  during  ALGM  TECHEVAL.   OPTEVFOR  conducts  what  amounts  to  an 
airborne  Reliability  Demonstration  [Ref.  19]  during  its  evaluation  of 
ALGM  operational  effectiveness  and  suitability.   The  redundancy  problem 
occurs  because  the  agencies  cited  feel  that  must  repeat  or  independently 
conduct  reliability  test  programs  to  verify  compliance  with  specifi- 
cation minimum  acceptable  MTBFs  or  probabilities  of  success  at  the 
stated  confidence  levels  even  though  the  contractor  may  have  success- 
fully done  so.   This  can  require  an  extraordinary  amount  of  time 
(schedule) ,  a  significant  commitment  of  aircraft  and  manpower,  and  a 
very  large  number  of  ALGM  test  assets. 

e.  Management  of  Diverse  Activities  and  Functions 

The  Separate  Test  Programs  in  Series  approach  severely 
taxes  the  matrix  management  approach  which  ALGM  Program  Managers  are 
compelled  to  use  as  a  matter  of  policy.   In  the  current  environment 
of  the  shrinking  dollar  and  fewer  new  programs,  competition  between 
field  activities  for  T&E  work  is  strong  and  the  feeling  of  the  need 
to  perpetuate  activities  and  capabilities  is  pervasive  throughout  the 
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Navy's  T&E  community.   The  competition  between  Navy  activities  within 
DoD's  Major  Range  and  Test  Facility  Base  (e.g.,  PMTC,  NATC,  and  NWC) 
creates  borderline  adversary  relationships  that  require  careful 
handling  on  the  part  of  the  Program  Manager  or  T&E  coordinator  to 
assure  successful,  timely  execution  of  the  T&E  program. 

Another  facet  of  this  competitive  situation  is  that  DD(T&E) 
is  actively  pursuing  reduction  of  unwarranted  duplication  of  facilities 
and  functions  among  facilities  which  constitute  the  Major  Range  and 
Test  Facility  Base  (MRTFB) [Ref .  3] . 
3 .   Other  Problems 

There  are  several  problems  outside  the  five  basic  areas  identi- 
fied in  section  IVA.l  which  also  impact  the  Program  Manager  in  his 
attempt  at  planning  and  implementing  an  adequate  T&E  program  using 
the  Separate  Test  Programs  in  Series  approach.   These  other  problems 
influence  the  T&E  program  as  follows: 

a.  If  the  Separate  Test  Programs  in  Series  approach  is  followed, 
with  little  or  no  interplay  between  DT&E  and  OT&E  while  the  program  is 
stretched  out,  appropriate  attention  may  not  be  directed  toward  main- 
tenance of  the  capability  to  meet  the  threat.   The  ALGM  may  satisfy 

the  specification  requirements  but  when  subjected  to  more  stringent 
OT&E  tests,  particularly  OPEVAL,  may  be  found  inadequate  to  meet  the 
evolving  threat. 

b.  With  this  approach  (Separate  Test  Programs  in  Series) , 
stretchout  of  the  schedule  and  consequent  dollar  impact  means  that 
there  will  be  a  great  deal  of  pressure  on  OPTEVEOR  to  expedite  OPEVAL 
because  of  the  rapidly  approaching  IOC  date.   There  will  be  entreaties 
to  compromise  test  objectives  to  conserve  assets  and  shorten  the  time 
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frame  and  somehow  soften  the  dollar  crunch.   This  same  situation  will 
also  make  it  extremely  difficult  to  plan  and  execute  the  FOT&E  effort. 

c.  ALGM  specifications  are  still  stated  in  terras  of  engineer- 
ing design  requirements  and  performance  parameters  measurable  in  the 
laboratory  which  may  not  be  relevant  to  an  operational  commander, 
according  to  OPTEVFOR  representatives.   Without  specifications  stated 
in  operational  terms,  it  is  difficult  for  OPTEVFOR  to  translate  speci- 
fication requirements  into  operational  requirements  and  evaluate  test 
results  accordingly.   The  Commission  on  Government  Procurement  [Ref.  11] 
observed  this  same  problem  and  suggested  that  once  the  engineering 
specifications  are  validated  in  the  DT&E  effort,  the  IOT&E  effort  should 
be  directed  toward  determining  whether  they  have  operational  value  or 
not. 

d.  According  to  a  DD(T&E)  staff  member  when  interviewed  re- 
cently, DD(TSE)  places  little  reliance  on  DT&E  test  data,  including 
TECHEVALs,  because  of  the  lack  of  independence  from  the  Developing 
Agency  and  because  of  contractor's  and  the  Program  Manager's  program 
advocacies  reflected  in  their  interpretation  of  test  data  and  presenta- 
tion of  results;  i.e.,  need  to  sell  the  program.   They  (DD(TSE))  feel 
that  OPTEVFOR  is  objective  and  unbiased  and  that  operational  testing 

is  the  prime  valid  measure  of  an  ALGM's  worth  and  utility  and  this  en- 
ables them  to  provide  the  DSARC  with  objective  unbiased  data  for  the 
decision  making  process.   DD(T&E)  feels  however  that  OPTEVFOR  needs 
more  analytical  and  technical  expertise,  citing  their  past  difficulty 
in  overall  test  planning  and  design  of  experiment  and  pointing  to  a 
future  need  for  the  ability  to  perform  simulations  which  are  going  to 
have  to  play  a  larger  role  in  future  TSE  programs  because  of  escalat- 
ing ALGM  asset  costs  and  test  costs. 
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4.   Good  Features 

Good  features  of  the  Separate  Test  Programs  in  Series  approach 
are  as  follows: 

a.  The  competition  created  between  T&E  activities  within  the 
Navy  T&E  community,  while  hard  on  the  personnel  involved,  may  yield  a 
more  efficient,  technically  competent  Major  Range  and  Test  Facility 
Base,  because  it  may  at  least  force  internal  self-examination  of  capa- 
bilities, costs,  and  personnel,  leading  to  reduction  of  duplication. 

b.  DD(TSE)  places  more  emphasis  and  reliance  on  operational 
test  results;  therefore,  with  the  potential  for  more  data  regarding  the 
ALGM's  effectiveness  and  suitability,  the  DSARC  decision  process  may  be 
easier  with  less  risk.  The  other  potential  attribute  is  that  more  time 
is  available  for  proofing  and  debugging  software  so  that  it  is  possibly 
less  of  a  problem  when  initially  deployed. 

3.   COMBINED  TESTING 

DoDD  5000.3  states  that,  "development  testing  and  early  phases  of 
operational  testing  may  be  combined  where  separation  would  cause  delay 
involving  unacceptable  military  risk,  or  would  cause  an  unacceptable 
increase  in  the  acquisition  cost  of  the  system"  [Ref .  6] . 

1.   Management  Approach 

The  latter  portion  of  the  quote  from  DoDD  5000.3  would  seem  to 
provide  seme  relief  to  the  Program  Manager  when  faced  with  inputs  to 
the  TEMP  from  the  T&E  participants  which  appear  similar  to  those  dis- 
cussed under  Separate  Test  Programs  in  Series;  i.e.,  separately  con- 
ducted duplicative  and  redundant  test  programs  that  can  cause  an  unac- 
ceptable increase  in  the  ALGM's  acquisition  cost.   If  this  problem  is 
recognized  early  enough  in  the  T&E  planning  stages,  a  viable  management 
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alternative  for  the  Program  Manager  is  to  strive  for  a  Combined  Test- 
ing approach  through  the  medium  of  the  TEMP  preparation  and  review 
cycle.   This  discussion  assumes  the  Program  Manager  is  faced  with  just 
such  a  situation,  recognizes  it  sometime  prior  to  Milestone  II  and 
attempts  to  rectify  the  projected  schedule  and  cost  growth  through 
combining  elements  of  DT&E  and  OT&E.   The  proposed  elements  of  the 
TEMP  (i.e.,  Contractor  Test  Program,  TECHEVAL  Program,  and  OT&E  Program) 
are  essentially  the  same  as  presented  in  the  preceding  section  VI  A. 

OPNAVINST  3960.10  permits  combined  testing  but  states  that,  "the 
final  sub-phases  of  DT-III  and  OT-III  (TECHEVAL  and  OPEVAL)  will  not 
normally  be  combined".   While  this  appears  to  be  a  significant  hindrance, 
the  major  obstacle  for  the  Program  Manager  to  overcome  is  the  fact  that 
he  and  COMOPTSVFOR  are  jointly  responsible  for  preparation  of  the  TEMP 
and  if  they  cannot  achieve  agreement  the  problem  is  passed  to  the  OPNAV 
program  sponsor  via  OP-983.   The  Program  Manager  states  the  areas  of 
disagreement  when  the  TEMP  is  submitted  and  COMOPTEVFOR  responds,  in 
writing,  with  the  rationale  for  those  areas  where  there  appears  to  be 
disagreement.   Due  to  the  Navy's  command  structure,  with  COMOPTEVFOR 
being  double-hatted  and  filling  the  position  of  DCNO  for  T&E,  the  Pro- 
gram Manager  would  appear  to  be  in  a  disadvantageous  position  when 
taking  exception  to  the  OT&E  program. 

TECHEVAL  is  an  equally  difficult  area  for  the  Program  Manager 
to  modify  or  waive  because  of  NAVAIR  standard  operating  procedures. 
With  few  exceptions,  ALGM  Program  Managers  find  it  almost  impossible 
to  modify  the  approach  to  ALGM  TECHEVAL  historically  taken  by  PMTC, 
Point  Mugu,  because  the  NAVAIR  T&E  community  looks  upon  this  as  its 
"independent"  evaluation  of  the  system. 


73 


The  most  likely  candidates  for  combining  tests  appear  to  be 
those  ALGM  system  level  tests  where  aircraft  are  involved  such  as, 
system  compatibility  and  integration  tests  (ALGM,  launcher,  aircraft, 
avionics) ,  captive  flight  performance  and  reliability  tests,  missile 
launch  and  free  flight  tests.   The  aforementioned  tests  are  commonly 
performed  in  the  DT&E  phase  by  the  contractor  (though  the  contractor 
rarely  operates  the  aircraft)  and  the  TECHEVAL  activity,  and  during 
OT&E  by  OPTEVFOR.   If  tests  are  combined  and  replications  avoided, 
T&E  cost  growth  and  schedule  problems  present  with  the  Separate  Test 
Programs  in  Series  should  be  significantly  reduced. 
2.   Applicable  T&E  Management  Problems 

In  addition  to  those  problems  alluded  to  in  the  preceding  para- 
graphs, a  number  of  significant  problems  remain,  primarily  in  the 
areas  of  planning  and  resource  and  schedule  constraints.   Management 
of  diverse  activities  and  functions  and  proliferation  of  requirements, 
as  previously  discussed,  will  continue  to  be  a  problem  but  do  not 
require  further  amplification  here.   The  problem  of  redundant  testing 
will  be  ameliorated  to  some  extent  by  the  combined  testing  approach, 
a.   Planning 

The  immediate  problem  in  planning  for  combined  testing 
stems  from  the  OPNAVINST  3960.10  prohibition  on  combining  TECHEVAL  and 
OPEVAL,  since  these  test  programs  appear  to  be  quite  similar.   This 
prohibition  relates  to  the  basic  purpose  of  DT&E  and  OT&E,  i.e.,  DT&E 
is  designed  to  assure  that  the  contractual  specifications  have  been 
met  and  OT&E  determines  whether  the  system  fulfills  the  desired  func- 
tions in  an  operational  environment.   OPTEVFOR  also  contends  its  evalua- 
tion is  objective  and  unbiased,  whereas  TECHEVAL  is  not  because  PMTC 
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is  part  of  the  Developing  Agency  (NAVAIR) .   The  Program  Manager  and 
other  T&E  planners  must  keep  in  mind  that  when  combined  testing  is  con- 
ducted, OPNAVINST  3960.10  requires  that  the  necessary  test  conditions 
and  test  data  required  by  both  the  DoD  component  developing  agency  and 
the  OT&E  agency  must  be  realized.   This  may  be  difficult  to  accomplish 
because  DT&E  tests  are  usually  structured  to  hold  many  parameters  con- 
stant, isolate  others,  and  allow  measurement  of  specific  quantities 
of  interest.   COMOPTEVFOR  says  this  is  usually  not  possible  in  OT&E  and 
it  is  often  not  even  possible  to  specify  what  OPTEVFOR  wants  to  measure. 
The  objective  is  often  to  create  conditions  as  close  to  combat  condi- 
tions as  possible  and  watch  what  happens  [Ref .  13] . 

A  significant  problem  to  the  T&E  planners  (with  DT&E  and 
OT&E  combined)  will  be  resolution  of  the  criteria  for  determining  whether 
tests  have  been  successful  or  not,  i.e.,  agreement  on  test  objectives 
and  technical  measures  of  effectiveness  will  be  difficult  to  achieve 
and  whether  or  not  to  proceed  to  the  next  test  or  series  of  tests  may 
be  quite  controversial.   The  Commission  on  Government  Procurement  [Ref. 
11]  recognized  this  situation  could  exist  and  addressed  the  problem  of 
applicability  of  engineering  specifications  to  the  operational  environ- 
ment by  recommending  some  form  of  early  OT&E  to  make  sure  that  engineer- 
ing specifications  have  operational  value. 

Finally,  OPNAVINST  3960.10  requires  that  the  OT&E  agency 
"insure  that  the  combined  test  is  so  planned  and  executed  as  to  provide 
the  necessary  operational  test  information;  participate  actively  in  the 
test;  and  provide  separate  evaluation  of  the  resultant  operational  test 
information" .   The  planning  and  execution  may  be  the  most  difficult 
aspect  to  assure.   OPTEVFOR  indicated  that  Program  Managers  seldom  in- 
volve them  early  in  the  program,  particularly,  in  the  early  planning, 
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but  said  the  situation  is  improving.   The  major  problem  would  appear  to 
be  the  planning  for  the  execution  of  the  test.   OPTEVFOR  stresses  an 
uncontrolled  operational  type  of  environment  with  operational  personnel, 
i.e.,  people  not  specially  trained,  not  test  pilots,  not  technicians  or 
engineers,  but  like  typical  officers  and  men  in  the  fleet. 

b.   Resource  and  Schedule  Constraints 

T&E  resources  for  combined  testing  are  of  special  concern. 
Test  facilities  and  ranges  that  would  be  required  for  ALGM  combined 
testing  are  primarily  R&D  oriented.   They  possess  good-to-excellent  in- 
strumentation with  a  high  degree  of  precision  in  measurement,  but  have 
very  little  flexibility  with  regard  to  the  environment  or  method  of 
application,  i.e.,  not  flexible  in  terms  of  application  of  operational 
tactics.   The  Blue  Ribbon  Panel  noted  problems  in  1970  [Ref.  8],  many 
of  which  are  still  with  us  today.   Most  of  the  facilities  and  ranges 
with  state-of-the-art  capability  (i.e.,  maneuvering  room  and/or  target 
complexes)  are  in  great  demand  from  a  scheduling  standpoint  just  to 
satisfy  development  (R&D)  requirements. 
3.   Other  Problems 

Several  facets  of  the  development  process  and  DT&E  make  a  com- 
bined DT&E  and  OT&E  evaluation  very  unlikely.   The  very  nature  of  the 
evolution  of  the  ALGM  product  through  DT&E  (building  block  testing 
from  components  to  systems) ,  limited  design  disclosure  documentation 
to  support  it,  and  the  evolutionary  philosophy  itself  (design-test-re- 
design-retest)  are  inhibiting  factors.   Finally,  OPTEVFOR  finds  that 
test  objectives  for  development  hardware  tests  are  usually  very  limited 
and  contractors  and  Program  Managers  are  usually  unwilling  to  test 
development  hardware  under  operational  conditions  until  late  in  the 
program. 
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4.   Good  Features 

The  obvious  good  feature  of  the  combined  testing  approach  is 
the  potential  opportunity  to  conserve  the  T&E  schedule  and  optimize  the 
utilization  of  ALGM  and  test  facility  assets,  thus  improving  the  pro- 
gram's cost  posture.   Another  facet  is  the  requirement  for  an  early 
positive  involvement  of  the  OT&E  forces  to  assure  compatibility  of  the 
ALGM  engineering  specifications  with  the  operational  environment  in 
which  the  weapon  will  have  to  perform. 

Finally,  the  Defense  Science  Board  observed  in  its  1977  report 
[Ref .  15]  that  a  means  should  be  sought  to  promote  interaction,  par- 
ticularly feedback  from  the  OT&E  to  the  developer.   Combined  testing 
would  provide  a  very  strong  vehicle  for  this  type  of  interaction.   The 
Defense  Science  Board  concluded  "interaction  among  development  test  and 
evaluation  and  close  contact  with  the  user  pays  very  important  dividends 
in  terms  of  money,  time,  and  operational  suitability". 

C.   INTEGRATED  TEST  PROGRAM 

Another  alternative  available  to  the  Program  Manager  for  the  ALGM 
T&E  is  the  Integrated  Test  Program  approach.   The  Integrated  Test  Pro- 
gram approach  features  performance  of  that  amount  of  testing  necessary 
to  confirm  compliance  with  specification  requirements  and  verify  opera- 
tional effectiveness  and  suitability.   Close  coordination,  interface 
and  pooling  of  information/data  by  all  T&E  participants  is  necessary 
to  cumulatively  build  the  net  data  base  with  each  successive  program 
phase.   Properly  executed,  the  Integrated  Test  Program  approach  mini- 
mizes ALGM  and  other  asset  requirements.   The  Integrated  Test  Program 
differs  from  Combined  Testing  in  that  the  DT&E  and  OT&E  testing  is  not 
conducted  together  or  simultaneously  but  is  conducted  in  a  fashion 
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similar  to  the  Separate  Test  Programs  in  Series,  exclusive  of  the  repeti- 
tion and  redundancy  and  with  an  operational  flavor  provided  by  early, 
continuing  OT&E  participation. 
1.   Management  Approach 

The  intent  of  the  Integrated  Test  Programs  approach  is  to  work 
within  the  requirements  and  guidelines  of  OPNAVINST  3960.10  while 
vigorously  controlling  costs  and  schedule  and  because  of  this,  much  of 
the  Management  Approach  information  relative  to  the  Separate  Test  Pro- 
grams in  Series  in  applicable.   The  descriptions  of  DT&E,  OT&E  and 
their  respective  phases  are  still  apropos  and  utilization  of  contractors, 
participating  field  activities,  and  other  support  activities  is  still 
required. 

The  early  stages  of  the  program,  prior  to  Milestone  I,  are 
critical  to  the  success  of  the  Integrated  Test  Program.   It  is  impera- 
tive that  the  Program  Manager  or  his  designee  for  T&E,  enlist  the 
support  and  active  participation  of  all  T&E  participants  in  the  prepara- 
tion of  ALGM  specifications  appropriately  couched  in  operational  termi- 
nology, followed  by  preparation  of  the  TEMP  for  CNO  approval  just  prior 
to  Milestone  I.   The  Program  Manager  must  make  it  clear  to  all  T&E 
participants  that  repetition  and  redundancy  are  not  usually  conducive 
to  a  timely  and  cost  effective  T&E  program.   He  must  emphasize  that 
testing  is  required  to  have  an  operational  flavor  beginning  in  the 
early  stages  and  will  be  building  block  in  form,  with  data  to  be  accumu- 
lated through  successive  stages  of  the  program.   Having  laid  this 
groundwork,  the  three  basic  test  programs  of  the  Integrated  Test  Program 
appear  as  follows : 
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a.   Contractor  Program 

DT-I  through  DT-III  as  described  in  section  III  C 1,  i.e., 
the  building  block  testing  approach  from  major  component  to  ALGM  section 
and  on  to  ALGM  level  culminating  with  Contractor  Demonstration  Tests, 
including: 

(1)  Part  and  component  qualification,  assembly,  and  sub- 
system tests  (accelerated  life,  overstress,  and  design 
evaluation  and  qualification) ,  sub-system  and  section 
integration  and  compatibility  tests. 

(2)  ALGM  system  level  performance  tests  on  the  ground 

(chamber  tests) ,  on  board  the  aircraft,  and  in  the 
captive  carry  mode  concluding  with  several  missile 
launches  from  aircraft, 

(3)  reliability  demonstration  for  compatibility  with  speci- 
fication requirements  (ground  and  airborne  testing  for 
captive  carry  MTBF,  launch  and  free  flight  to  target 
reliability)  at  the  required  confidence  levels, 

(4)  maintainability  demonstration  for  compatibility  with 
specification  requirements, 

(5)  environmental  and  safety  qualification  (including 
electromagnetic  compatibility)  to  verify  compliance 
with  specification  design  requirements. 

OPTEVFOR  and  TECHEVAL  personnel  should  support  the  Program  Manager  and 
contractor  in  establishing  specification  requirements  responsive  to 
the  threat  with  user  inputs  which  are  representative  of  the  operational 
environment.   After  early  verification  of  adequacy,  continuing  effort 
must  be  expended  to  maintain  compatibility  with  the  operational 
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environment.   OPTEVFOR  and  TECHEVAL  should  support  the  Program  Manager 
and  his  participating  field  activity  Technical  Manager  in  reviewing 
for  government  approval  all  Contractor  Demonstration  Test  (CDT)  plans, 
test  procedures,  and  test  reports  and  should  actively  participate  in 
monitoring  contractor  CDTs  to  assure  technical  compliance  and  opera- 
tional adequacy. 

b.  TECHEVAL  Program 

Performed  during  the  final  sub-phase  of  DT-III  and  includes: 
ALGM  system  level  performance  tests  both  on  the  ground  (chamber  tests) , 
on  board  the  aircraft,  and  in  the  captive  carry  mode  concluding  with 
several  launches,  reliability  data  acquisition  and  evaluation  as.  part 
of  the  ALGM  ground  and  airborne  testing,  sufficient  to  confirm  ALGM 
adequacy  and  readiness  for  OPEVAL  but  not  a  repeat  of  the  Contractor 
Demonstration  Test.   The  intent  is  to  build  on  the  Contractor  Demonstra- 
tion Test  reliability  data  base,  ALGM  configuration  permitting.   Main- 
tainability and  supportability  should  be  evaluated  in  concert  with  the 
testing  described  above.   It  is  not  necessary  to  repeat  the  Contractor's 
maintainability  demonstration  at  this  stage.   Implementation  of  the 
foregoing  TECHEVAL  results  in  significant  ALGM  program  cost  savings 
over  other  T&E  alternatives  through  conservation  of  T&E  resources  and 
schedule  improvement. 

c.  OT&E  Program 

OT-II  and  OT-III,  as  described  in  section  II C  2,  is  per- 
formed culminating  with  OPEVAL,  which  includes:   ALGM  system  level  per- 
formance tests  on  board  the  aircraft,  on  the  ground  and  in  the  captive 
carry  mode  concluding  with  several  missile  launches,  airborne  data 
acauisition  and  evaluation  of  captive  carry  MTBF,  and  launch  and  free 
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flight  reliability  data  as  part  of  the  ALGM  ground  and  airborne  testing. 
This  should  be  sufficient  to  confirm  operational  effectiveness  and  suit- 
ability and  when  coupled  with  the  accumulated  applicable  Contractor 
Demonstration  Test  and  TECHEVAL  data,  should  complete  the  development 
characterization  of  the  ALGM  and  confirm  its  compliance  with  specifica- 
tion requirements  and  readiness  for  production.   Suitable  confidence 
limits  should  be  employed  and  appropriate  truncation  techniques  applied 
as  necessary  under  the  guidance  of  competent  statistical  support  help 
to  maintain  statistical  rigor. 

Maintainability,  supportability  and  other  logistics  related 
ALGM  system  characteristics  as  well  as  compatibility,  operability,  human 
factors,  and  training  adequacy  should  be  evaluated.   Tactics  development 
and  refinement  should  continue  as  necessary. 

d.   FOT&E  Program 

OT-IV  is  performed  as  described  in  section  IIIc.3  and  under 
the  Separate  Test  Programs  in  Series  approach. 
2.   Applicable  T&E  Management  Problems 

Many  of  the  problems  facing  the  Program  Manager  are  similar  to 
those  to  be  encountered  in  the  prosecution  of  any  T&E  program.   However, 
the  nature  of  the  Integrated  Test  Program  and  demands  it  makes  on  the 
Program  Manager  do  create  some  unique  situations.   The  spectrum  of  prob- 
lems identified  in  section  IV  have  some  degree  of  applicability  but 
redundant  testing  with  duplication  of  facilities ,  personnel,  and  their 
related  costs  is  clearly  reduced  in  significance. 

a.   Planning 

The  Program  Manager  will  have  to  exercise  strong  powers  of 
persuasion  to  get  the  program  T&E  participants  together  with  users  early 
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and  obtain  the  cooperation  needed  for  planning  vital  to  the  structuring 
of  the  program  budget  and  development  of  the  baseline  TEMP.   This  early 
cooperative  effort  will  have  to  be  sustained  throughout  the  program  to 
provide  the  impetus  for  implementation  of  the  T&E  program  and  for  main- 
tenance and  updating  of  the  TEMP  and  the  T&E  portions  of  the  DCP.   The 
Program  Manager  will  have  to  plan  the  acquisition  of  the  ALGM  design 
disclosure  documentation  package  carefully  and  provide  for  configuration 
management  in  a  timely  fashion.   The  ALGM  design  disclosure  documenta- 
tion package  evolves  with  the  design  and  is  necessary  to  define  and  con- 
trol hardware  configurations,  and  changes  thereto,  while  undergoing  T&E. 
This  will  support  and  lend  credibility  to  the  test  programs  by  assuring 
knowledge  of  what  was  tested  and  will  ease  the  evaluation  of  the  effec- 
tiveness of  changes.   Also,  if  hardware  configurations  are  known  and  con- 
trolled, development  of  a  credible  test  data  base  will  be  made  easier. 
This  will  be  a  difficult  problem  to  overcome  because,  in  the  past,  accord- 
ing to  NAVAIR,  contractor,  and  participating  field  activity  personnel, 
acquisition  of  the  ALGM  design  disclosure  data  package  has  often  been 
mishandled  and  not  fully  supported  by  Navy  program  sponsors  and  the 
Developing  Agency. 

b.   Proliferation  of  Requirements 

The  Program  Manager  will  still  have  to  contend  with  problems 
due  to  requirements  unique  to  certain  participating  field  activities, 
test  facilities,  ranges  and  other  agencies  as  indicated  in  section  V  A  2b. 
A  strong  early  planning  effort  with  full  participation  and  cooperation 
from  the  T&E  community  providing  clear  delineation  of  the  program  to  be 
conducted  and  participant's  responsibilities  with  related  budgets,  should 
aid  in  the  control  process.   The  Program  Manager  requires  the  assistance 


82 


of  higher  Navy  management  and  OSD  in  limiting  the  number  of  "special" 
management  reviews  of  the  T&E  program  as  well  as  other  aspects  of  the 
program.   This  should  be  identified  as  an  issue  early  in  the  program 
and  brought  to  higher  management  attention  during  early  program  reviews 
and  when  the  T&E  program  is  presented  for  approval  at  Milestone  I.   The 
NMARC  report  recognized  this  as  a  significant  problem  for  the  Program 
Manager,  stating  that,  "his  program  is  reviewed  at  various  levels  up 
through  DoD  by  offices  with  little  or  no  accountability.   Hence,  he  can 
be  "second  guessed"  by  all  concerned,  yet  he  is  the  specific  individual 
who  must  assign  his  assets  in  the  most  effective  manner  to  execute  the 
overall  program"  [Ref .  16] . 

c.   Resource  and  Schedule  Constraints 

Scarcity  of  ALGM  assets  and  test  facilities,  possibly  impact- 
ing T&E  schedules,  should  be  no  more  severe  than  under  the  other  T&E 
approaches  and  may  actually  be  alleviated  by  the  Integrated  Test  Program. 
Fewer  total  ALGM  assets  and  hopefully,  fewer  facilities  are  involved  but 
specific  targets  and/or  target  complexes  will  be  in  demand  and  range 
instrumentation  taxed.   Maintenance  problems  may  result  because  of  higher 
utilization  rates. 

Limited  test  flexibility  and  ability  to  conduct  operational 
type  resting  on  what  primarily  are  still  R&D  oriented  ranges  will  in- 
hibit the  user-evaluator 's  IOT&E  effort.   Bowes  [Ref.  9]  also  found 
that  a  major  portion  of  the  physical  resources  utilized  by  OPTEVFOR  for 
its  operational  evaluation  are  controlled  by  the  Developing  Agency.  The 
Integrated  Test  Program  approach  provides  the  mechanism  for  the  develop- 
ment of  plans  for  the  utilization  of  these  limited  resources  by  OPTEVFOR 
and  the  DT&E  agencies.   Lack  of  control  of  the  major  portion  of  the 


physical  resources  required  for  OT&E  (instrumentation,  ranges,  data 
processing  centers,  simulators,  and  test  aircraft)  appears  to  reduce 
the  independence  of  OPTEVFOR. 

d.  Redundant  Testing 

As  previously  states  in  section  VC,  redundant  testing  will 
be  virtually  eliminated  as  a  significant  problem  in  the  ALGM  T&E  pro- 
cess by  implementation  of  the  Integrated  Test  Approach.   Performance, 
reliability  and  maintainability  testing  will  not  consist  of  repeated 
lengthy  and  costly  demonstrations  of  compliance  with  design  require- 
ments but  will  be  cumulative  in  nature,  under  conditions  reflecting 
operational  conditions  agreed  to  in  advance  by  all  T&E  participants. 
Credibility  of  the  test  base  will  have  to  be  assured  through  proper  test 
design  and  control,  as  well  as  early  implementation  of  configuration 
management  and  a  strong  failure  analysis  and  corrective  action  program. 
Implementation  of  the  foregoing  demonstrates  a  well  thought  out, 
plausible  T&E  approach,  with  reasonable  test  objectives  and  appropriate 
numbers  of  assets  of  known  configuration  with  other  necessary  resources 
to  progressively  attain  T&E  objectives. 

e.  Management  of  Diverse  Activities  and  Functions 

The  Program  Manager  will  be  severely  tested  as  will  the 
Navy's  matrix  organization  in  the  implementation  of  the  Integrated 
Test  approach.   The  Program  Manager  must  strive  for  early  cooperation 
and  acceptance  by  the  participating  NAVAIR  design  groups  and  test 
facilities,  participating  field  activities,  OPTEVFOR,  and  the  contractor, 
The  contractor  may  initially  be  balky  when  faced  with  the  possibility 
of  early  exposure  to  ALGM  test  vehicles  to  at  least  a  synthesized 
operational  environment.   Contractors  and  Navy  Program  Managers  fear 
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that  early,  visible  failures  under  anything  approaching  operational 
conditions,  could  result  in  program  cancellation.   Incorporation  of 
operational  requirements  in  design  specifications  will  be  difficult, 
requiring  close  coordination  between  all  design  groups  and  T&E  program 
participants.   The  difficulty  arises  because  of  the  limited  success 
to  date  in  translating  operational  requirements  into  terminology  and 
units  for  inclusion  in  specifications  for  test  and  measurement  [Ref .  20] . 

With  regard  to  the  TECHEVAL  portion  of  the  T&E  program,  the 
Program  Manager  may  receive  considerable  opposition  from  within  the 
NAVAIR  T&E  community  and  from  PMTC  because  of  the  proposed  reduced  scope 
of  TECHEVAL.   However,  when  interviewed  recently,  the  incoming  AIR-06 
(formerly  the  PMA  for  ARtl  missiles,  possessing  major  acquisition  exper- 
ience) indicated  that  in  his  opinion,  too  much  testing  had  been  done  in 
the  past  for  testing's  sake  and  that  the  T&E  program  has  to  be  tailored 
to  the  overall  weapons  acquisition  process.   He  also  suggested  that 
early  planning  was  key  and  would  enhance  the  program's  stature  with  OSD. 
3.   Other  Problems 

The  Program  Manager  will  have  to  assure  early  efforts  on  the 
part  of  the  potential  ALGM  users,  OPTEVFOR,  NAVAIR  and  contractor  design 
groups,  in  the  development  of  test  envelopes  with  respect  to  the  mission 
profile  from  which  T&E  test  plans  will  evolve.   Mechanical  model  ALGMs 
will  have  to  be  used  for  aircraft  captive  carry,  laboratory  environmental 
determination  efforts,  and  compatibility  with  designated  and  potential 
user  aircraft. 

Another  concern  for  the  Program  Manager  will  be  the  need  to  make 
DD(TSE)  aware  of  the  intended  approach,  prior  to  Milestone  I  if  possible, 
and  take  the  necessary  steps  to  retain  his  understanding  and  concurrence 


85 


throughout  the  program.   PMA-242  (the  Anti-Radiation  Missile  Program 
Manager) ,  in  an  interview,  indicated  that  periodic  detailed  briefings 
to  DD(TSE)  have  served  to  promote  an  understanding  and  acceptance  of 
lower  level  testing,  simulations,  and  the  fact  that  the  totality  of 
the  worth  of  an  ALGM  system  should  not  be  based  on  free  flight  per- 
formances alone.   DD(TSE)  may  also  require  assurance  of  OPTEVFOR's 
independence  and  their  continued  ability  to  be  objective,  even  though 
involved  very  early  in  the  program  and  to  the  extent  required. 
4.   Good  Features 

Implementation  of  an  Integrated  Test  Program  affords  an  oppor- 
tunity to  the  ALGM  Program  Manager  to  alleviate  the  T&E  impact  on 
overall  program  costs  and  to  exert  better  schedule  control.   The  Defense 
Science  Board  indicated  "there  may  be  a  potential  to  reduce  the  large 
demands  of  reliability  and  maintainability  testing  if  the  various 
agencies  responsible  for  such  testing  could  develop  coordinated  and  in- 
tegrated test  plans.   That  is,  factory  chamber  testing,  developmental 
testing  under  operational  conditions,  and  operating  testing  all  provide 
opportunity  for  collecting  R&M  data"  [Ref .  15] .   The  Integrated  Test 
Program  approach  should  preserve  the  distinct  DT&E  and  OT&E  purposes 
and  assure  that  appropriate  personnel  are  doing  the  testing  required 
under  each  type  of  testing. 

Greater  involvement  of  OPTEVFOR  should: 

a.  help  to  alleviate  problems  with  regard  to  development  and 
review  of  incorporation  of  operational  requirements  into 
design  specifications  and  translation  of  these  requirements 
into  tests, 

b.  bring  consistency  in  terms  of  test  requirements,  test 
methods,  and  test  data  acquisition  and  interpretation, 
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c.  make  it  easier  to  stay  abreast  of  the  environment  in  which 
the  threat  will  be  encountered,  and 

d.  help  make  DT&E  test  results  more  credible  and  acceptable 
to  DD(T&E)  and  the  DSARC  committee. 

The  Integrated  Test  approach  should  also  help  to  alleviate 
Defense  Science  Board  concerns  with  regard  to  repetitive  and  redundant 
testing  and  NMARC  concerns  regarding  an  uneconomical  amount  of  testing 
and  testing  beyond  the  capability  of  the  hardware.   And  finally,  the 
Integrated  Test  approach  should  provide  an  auditable  trail  of  require- 
ments versus  performance  which  has  been  of  great  concern  to  both 
DD(TSE)  and  the  NMARC. 

D.   SYSTEM  LEVEL  FUNCTIONAL  TEST  PROGRAM 

The  last  alternative  or  approach  to  be  considered  is  that  of  the 
System  Level  Functional  Test  Program.   Proponents  of  this  method  require 
that  testing  be  accomplished  at  the  ALGM  system  level  on  the  assumption 
that  the  ALGM  system  and  all  of  its  parts  can  satisfactorily  demonstrate 
required  capabilities  and  be  qualified  as  a  unit.   This  approach  is 
usually  suggested  when  the  item  to  be  tested  is  not  very  complex,  as 
a  viable  alternative  in  a  crisis  situation,  or  as  a  "get  well"  remedy 
when  schedule  slippage  and  cost  growth  is  first  detected  in  DT&E.   The 
intent  in  the  testing  mcde  is  to  perform  as  few  tests  as  possible,  con- 
serve ALGM  test  assets,  have  the  shortest  possible  development  T&E 
schedule,  with  minimum  cost.   Unfortunately,  avionics  equipments 
("black  boxes")  are  most  often  cited  as  representative  of  hardware 
qualified  for  service  use  in  this  fashion  rather  than  existing  ALGMs ; 
also,  most  of  the  equipments  cited  were  developed  under  quick  response 
contracts  during  hostilities  in  southeast  Asia  and  were  few  in  number. 
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This  approach  is  an  extremely  optimistic  one.   It  would  appear,  on  the 
surface,  to  reduce  ALGM  asset  requirements,  provide  a  shorter  develop- 
ment schedule  than  any  of  the  others  considered,  and  therefore  be  cost 
effective. 

1.   Management  Approach 

The  management  approach  is  quite  similar  to  that  of  the  Separate 
Test  Programs  in  Series  or  Combined  Test  Programs  with  the  exceptions 
as  follows: 

a.  Contractor  Program 

The  contractor  portion  of  the  DT&E  phase  essentially  discards 
the  building  block  approach  to  hardware  development,  i.e.,  seeks  to 
avoid  the  individual  performance,  reliability,  and  environmental  quali- 
fication of  critical  parts,  components,  and  assemblies.   This  approach 
also  avoids  or  minimizes  sub-system  testing  and  tries  to  accomplish 
interface  evaluation  and  compatibility  verification  at  the  time  ALGM  or 
system  level  testing  is  conducted.   Performance,  reliability,  and  en- 
vironmental qualification  (i.e.,  Contractor  Demonstration  Tests)  are 
performed  at  ALGM  level,  to  ALGM  system  specifications. 

b.  TECHEVAL  Program 

The  TECHEVAL  is  planned  and  conducted  as  with  Separate  Test 
Programs  in  Series  (section  VA  1  b)  but  without  the  prior  testing  and 
verification  foundation  usually  acquired  through  the  contractor  program. 

c.  OT&E  Program 

OT&E  is  planned  and  conducted  as  previously  indicated  (i.e., 
Separate  Test  Programs  in  Series,  section  VA  1  c) . 
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2.   Applicable  T&E  Management  Problems 

a.  Planning 

Planning  is  extremely  limited  and  optimistic  to  begin  with 
(testing  at  ALGM  level  only  and  no  contingency  plans  for  failure)  and 
problems  which  occur  at  ALGM  system  level  are  much  more  difficult  to 
diagnose,  analyze,  and  correct.   Failures  and  related  delays,  redesigns 
and  corrective  action  assessments  with  retests  can  consume  significant 
amounts  of  time,  impact  schedule  and  costs  severely,  and  cannot  be 
planned  for. 

The  planning  problem  is  even  more  basic  than  the  previous 
paragraph  would  indicate.   In  today's  acquisition  environment,  the 
Program  Manager  and  his  staff  would  probably  encounter  problems  in 
arriving  at  an  agreed  upon  TEMP  approach  with  COMOPTEVFOR  and  doubtless 
would  experience  great  difficulty  in  getting  DD(T&E)  acceptance  and/or 
concurrence  to  this  approach  unless  there  were  unusual  mitigating  cir- 
cumstances, i.e.,  national  emergency,  significant  technological  gain 
that  might  be  lost  without  a  maximum  concurrency  type  of  approach, 
or  some  similar  situation  of  urgency. 

b.  Proliferation  of  Requirements 

Problems  here  are  essentially  the  same  as  with  Separate 
Test  Programs  in  Series. 

c.  Resource  and  Schedule  Constraints 

Even  with  the  limited  asset  approach  indicated,  certain 
major  components  and  sections  of  the  ALGM  will  require  at  least  limited 
qualification  and  man-rating  (proof  of  safe-for-launch  from  manned 
aircraft)  prior  to  introduction  into  ALGM  system  level  testing.   One 
shot  devices,  batteries,  fuzes,  warheads  and  propulsion  systems 
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(rocket  motors)  fall  into  this  category.   Until  the  limited  qualifica- 
tion is  accomplished,  system  level  testing  will  be  inhibited  signifi- 
cantly. 

With  the  optimistic  limited  asset  situation,  there  is  little 
margin  for  error.   There  could  be  a  severe  resource  problem  with  major 
schedule  impact,  as  a  result  of  failures,  that  could  virtually  bring 
the  program  to  a  standstill.   Without  adequate  resources  (ALGM  assets) , 
problem  evaluation,  corrective  action  determination  and  re- test  cannot 
proceed.   A  requirement  to  build  additional  hardware  (i.e.,  re-start 
fabrication  process)  could  effectively  cause  termination  of  the  program. 
Finally,  according  to  the  Defense  Science  Board  [Ref .  11] ,  hardware 
problems ,  a  shortage  of  funds  and  schedule  problems  inhibits  thorough 
risk  assessment  when  problems  occur.   Without  the  risk  assessment,  the 
Defense  Science  Board  says  the  DSARC  will  be  faced  with  complex  technical 
issues  requiring  a  detailed  review  and  may  not  permit  progression  to 
succeeding  milestones. 

d.  Redundant  Testing 

These  problems  will  essentially  be  the  same  as  encountered 
with  Separate  Test  Programs  in  Series  (section  V  A  2  d) . 

e.  Management  of  Diverse  Activities  and  Functions 

Accent  on  minimum  testing  may  create  problems  with  certain 
activities  and  functions  as  discussed  under  V  D  1  a;  otherwise,  problems 
will  be  similar  to  those  of  Separate  Test  Programs  in  Series. 
3.   Other  Prcblems 

It  is  virtually  impossible  to  stress  all  parts  in  an  ALGM  to 
system  design  limits  much  less  get  all  parts  to  function  as  they  would 
under  mission  conditions,  while  under  system  test  conditions,  due  to 
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testing  limitations  posed  by  laboratories,  facilities,  and  equipment. 
Therefore,  part,  component,  assembly,  sub-system  tests  leading  to  a  num- 
ber of  system  tests  is  the  optimum  way  to  proceed.   Even  with  the 
building  block  approach,  failures  occur,  but  with  testing  at  ALGM  level 
only  there  will  be  failures  that  will  be  extremely  difficult  to  fault 
isolate  and  some  may  even  go  undetected  until  subsequent  phases  of 
testing  where  the  true  cause  of  failure  may  be  masked. 

The  1974  Defense  Science  Board  report  [Ref.  14]  stated,  "Signi- 
ficant time  and  money  will  be  saved  if  each  component,  each  sub-system, 
and  the  full  system  are  all  tested  as  thoroughly  as  possible  in  the 
laboratory".   The  report  went  on  to  say,  "V7henever  field  testing  is  ex- 
pensive compared  with  laboratory  and  simulation  testing  (as  in  missile 
or  aircraft/missile  flight  testing) ,  such  testing  should  be  conducted 
primarily  for  verification  of  design  parameters  or  design  performance, 
rather  than  to  see  whether  or  not  a  particular  component  or  sub-system 
will  work" . 

4.   Good  Features 

As  seen  in  sections  V  D  2  and  V  D  3,  this  approach  suffers  from 
its  departure,  in  the  DT&E  phases,  from  the  proven  hardware  building 
block  approach  advocated  by  the  Defense  Science  Board  and  has  a  high 
probability  of  failure.   There  are  no  known  or  recognizable  features 
significant  enough  to  overcome  the  inherent  deficiencies  in  this  testing 
approach.   It  does  not  appear  to  provide  the  basis  for  a  sound  develop- 
ment approach  and  it  increases  the  acquisition  risks  greatly. 
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In  this  section,  four  of  several  possible  alternative  program 
management  approaches  to  T&E  programs  for  ALGMs  have  been  presented. 
Each  alternative  was  analyzed  in  terms  of  its  management  methodology, 
applicability  of  management  problems  identified  in  section  IV  to  each 
alternative,  and  other  problems  deemed  relevant  and  unique  to  that 
alternative.   Favorable  features  were  also  identified  as  part  of  the 
analysis  for  each  alternative.   These  analysis  findings,  in  terms  of 
positive  and  negative  features,  are  summarized  in  Table  I.   Any  one  of 
the  initial  three  alternatives  would  be  acceptable  given  the  proper 
circumstances  and  resources. 
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VI.   PROPOSED  TEST  PROGRAM  STRATEGY 

Bowes  [Ref.  9]  cited  less  than  optimal  utilization  of  the  Navy's 
T&E  resources  and  found  that  the  prevalent  belief  that  "more  testing  is 
better,  so  let's  continue  to  do  more  and  not  worry  so  much  about  the 
duplication"  was  a  simplistic  and  costly  solution  to  the  Navy's  TSE 
problem.  Changes  have  occurred  in  Navy  T&E  subsequently,  but  current 
OSD,  CNO,  and  NAVMAT  directives  and  instructions  have  not  been  imple- 
mented by  directives  and  instructions  from  the  Developing  Agency  most 
responsible  for  Navy  ALGM  development.   In  the  absence  of  specific 
promulgated  directives  and  guidelines  and  direct  ALGM  program  partici- 
pation by  AIR-06  personnel,  Program  Managers  are  still  faced  with  the 
beliefs  and  perceptions  cited  by  Bowes. 

Consideration  of  the  alternative  approaches  to  conduct  of  the  TSE 
program  for  ALGMs  in  section  V  indicates  that,  on  balance,  the  Integrated 
Test  Program  approach  provides  the  best  opportunity  for  the  ALGM  Program 
Manager  to  conduct  an  adequate,  cost-effective,  T&E  program  while  work- 
ing within  the  requirements  and  guidelines  of  OPNAVINST  3960.10. 
Figure  7  presents  the  organizations,  and  their  respective  functions, 
which  must  cone  together  early  in  the  program  to  assist  the  Program 
Manager  in  developing  and  implementing  the  ALGM  T&E  program. 

The  purpose  of  this  section  is  to  build  upon  the  Management  Approach 
portion  of  section  V  C  and  propose  a  "Test  Program  Strategy"  which  will 
aid  ALGM  Program  Managers  in  the  development  and  implementation  of  an 
Integrated  Test  Program.   The  proposed  strategy  will  preserve  the  DT&E 
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and  OT&E  purposes  while  reducing  repetitive  and  redundant  testing,  pro- 
vide an  audi table  trail  of  requirements  versus  performance,  and  help 
to  bring  consistency  to  ALGM  testing  in  terms  of  test  requirements, 
test  methods,  and  test  data  acquisition  and  interpretation.   The  In- 
tegrated Test  Program,  to  which  the  proposed  strategy  is  applicable, 
is  as  depicted  in  section  V  C  1  and  consists  of  the  Contractor,  TECHEVAL, 
OT&E,  and  FOT&E  programs  as  outlined. 

A.   ELEMENTS  OF  THE  PROPOSED  STRATEGY 

The  proposed  strategy  to  be  employed  in  an  Integrated  Test  Program 
consists  of  consideration  and  implementation  of  a  number  of  activities 
under  the  major  ALGM  elements  of: 

1.  Planning 

2.  Types  and  Categories  of  Tests  and  Resource  Requirements 

3.  Responsibility  for  and  Conduct  of  Tests 

4.  Information  Flow,  Assessment  and  Reporting 

The  necessary  actions  to  implement  the  strategy  are  developed  in  subse- 
quent paragraphs . 
1.   Planning 

The  purpose  of  the  Integrated  Test  Program  approach,  as  it  is 
with  any  test  program,  is  to  perform  an  optimum  amount  of  testing  (i.e., 
that  amount  necessary  to  confirm  compliance  with  specifications  and 
operational  effectiveness  and  suitability)  and  no  more.   Repetition, 
duplication,  and  redundancy  are  to  be  minimized.   Hence,  planning  has 
been  emphasized  as  a  critical  element  in  any  T&E  program  (section  III  H) 
but  as  pointed  out  in  section  V  C  2,  it  is  especially  critical  to  the 
Integrated  Test  Program  approach  because  of  the  very  close  coordination 
and  interface  required  of  different,  widely  dispersed  organizations  and 
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because  integration  or  pooling  of  test  data  is  necessary  to  successfully 
demonstrate  compliance  with  test  objectives  while  utilizing  minimum  ALGM 
assets  and  other  resources. 

Areas  of  consideration  where  actions  should  be  taken  to  improve 
early  and  follow-on  planning  and  also  to  increase  the  overall  test  and 
evaluation  efficiency  are  as  follows: 

a.   Operational  Requirements  vs  Design  Specifications 

A  very  important  facet  of  the  early  planning  strategy  requir- 
ing user  participation  or  representation  is  the  development  of  operational 
or  functional  requirements  suitable  for  inclusion  in  design  specifications 
to  which  contractor  designed  and  built  ALGM  hardware  will  have  to  perform. 
These  requirements  become  the  basic  criteria  for  the  TSE  program  and  for 
acceptance  of  T&E  results.   Tactical  employment  of  the  ALGM  should  be 
considered  with  ample  resources  brought  to  bear  to  develop  a  mission  pro- 
file for  inclusion  in  the  specification.   The  mission  profile  should  re- 
flect ALGM  operating  duty  cycles  based  on  tactical  usage  and  should  also 
consider  the  peacetime  vs  wartime  scenario  where  expenditure  rates  effect 
the  number  of  cycles  (catapult  launch,  captive  carry,  and  arrest  landings) 
that  ALGMs  may  be  subjected  to.   The  mission  profile  should  also  reflect 
consideration  of  the  ALGM  launch  envelope,  aircraft  maneuvers,  cockpit 
workload,  and  interface  with  other  systems  on  the  user  aircraft.   Aircraft, 
launcher,  ALGM  interfaces  should  be  considered  and  user  experience  in 
these  areas  should  be  reflected  in  the  design  specifications.   Finally, 
the  design  and  operational  or  functional  requirements  must  be  translat- 
able into  test  requirements  which  are  agreed  upon  by  the  T&E  participants 
and  can  be  employed  in  DT&E  and  OT&E  phases  of  the  program  to  confirm 
ALGM  compatibility  with  design  requirements  and  military  worth  and 
utility. 
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b.  Test  Hardware  Considerations 

The  quantity  of  ALGM  test  assets  required  to  support  the 
T&E  program  and  the  amount  of  time  provided  for  successful  completion 
of  all  testing  is  critical  to  program  success.   Early  planning  must  be 
performed  to  provide  sufficient  time  and  resources  for  operational 
tests,  particularly  the  IOT&E  phase,  because  the  production  and  deploy- 
ment decision  is  based  upon  IOT&E  results.   Correspondingly,  later  in 
the  program,  the  Program  Manager  should  not  lose  sight  of  the  continuing 
need  for  test  articles.   There  have  been  circumstances  where,  for  various 
reasons,  the  technology  has  had  to  be  pushed  rapidly;  the  higher  risks 
involved  have  required  more  thorough  testing  and/or  greater  resources. 
These  programs,  usually  had  to  have  more  assets  and  have  been  expanded 
to  longer  times.   Typical  hardware  and  facility  requirements  for  an 
ALGM  program  are  addressed  in  VI  A  2. 

c.  Test  Facilities  and  Instrumentation  Requirements 

Before  Milestone  I,  the  test  facilities  and  instrumentation 
requirements  to  conduct  tests  should  be  identified  along  with  a  tenta- 
tive schedule  of  test  activities.   AIR-06  personnel  must  play  a  key  role 
in  this  effort  because  of  its  relationship  to  the  Navy's  Major  Range  and 
Test  Facility  Base  (MRTFB) .   They  must  keep  in  mind  that  OPTEVFOR  is 
totally  dependent  upon  others  for  their  test  facilities  and  instrumenta- 
tion and  that,  historically,  test  ranges  have  inhibited  operational  test- 
ing because  of  their  primarily  R&D  orientation  and  inherent  lack  of 
operational  flexibility.   The  capabilities  of  the  test  ranges  and  the 
adequacy  of  the  facilities  and  instrumentation  should  be  verified; 
alternative  approaches  considered  (i.e.,  other  ranges  should  be  identi- 
fied) and  the  need  for  instrumentation  improvements  or  changes  be 
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identified  early  in  the  program.   Range  and  instrumentation  limitations 
may  place  constraints  on  ALGM  flight  testing  and  targets  warrant  special 
consideration;  most  ALGMs  cannot  be  adequately  tested  without  targets 
that  realistically  simulate  the  threat.   Considerable  lead  time  may  be 
required  to  provide  the  required  target  support;  therefore,  target  re- 
quirements to  support  the  tests  must  be  identified  early, 
d.   Test  Configuration  Control 

Proper  ALGM  and  component  configuration  identification  and 
control  throughout  the  test  program  is  essential  to  the  creation  and 
maintenance  of  a  credible  test  data  base.   If  it  is  not  known  what  was 
tested,  the  test  results  are  not  too  significant.   Early  planning  will 
be  required  to  develop  the  appropriate  means  of  recording  ALGM  and/or 
component  configuration  (i.e.,  log  books  or  similar  methods)  and  for 
relating  the  configuration  to  test  results.   The  recording  means  must 
be  compatible  with  different  working  environments  and  users,  i.e., 
laboratory  and  test  range,  contractors,  participating  field  activities, 
and  OPTEVFOR.   Conditions  and  component  configuration  during  development 
tests  should  be  determined  by  the  primary  objectives  of  that  test. 

Whenever  a  non-operational  configuration  (e.g.,  ALGM  with 
inert  warhead  and/or  rocket  motor)  is  dictated  by  early  test  require- 
ments, tests  should  not  be  challenged  by  the  fact  that  the  configuration 
is  not  operational.   Where  tests  are  run  with  substitute  parts,  proce- 
dures should  provide  for  recording  the  fact  and  ensuring  that  necessary 
retesting  is  done  with  the  correct  components.   When  testing  is  delayed 
because  of  the  non-availability  of  critical  sub-system  components,  off- 
the-shelf  interim  components  may  be  used  as  substitutes  until  the  proper 
components  are  available.   As  long  as  the  off-the-shelf  components  can 
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function  acceptably  within  a  defined  range  of  interest,  the  rest  of  the 
system  can  be  tested,  thereby  facilitating  the  progress  of  the  test  pro- 
gram.  On  the  other  hand,  demonstration  and  acceptance  tests,  as  well 
as  tests  intended  to  evaluate  performance  under  operational  conditions, 
should  always  be  conducted  under  conditions  as  close  to  those  anticipated 
in  operations  as  possible,  including  present  ALGM  configuration. 

The  reliability  data  base  is  especially  susceptible  to  prob- 
lems created  by  poor  configuration  control.   To  pool  or  group  reliability 
data,  as  desired  with  the  Integrated  Test  approach,  the  hardware  tested 
must  be  from  a  homogeneous  population.   This  determination  of  homogeneity 
through  exercise  of  change  control  will  require  particular  care  once  ALGM 
prototype  and  pilot  production  hardware  becomes  available  for  testing 
and  demonstration  in  DT-III  and  OT-III.   If  change  control  is  ineffective 
or  major  design  changes  are  dictated,  there  may  be  no  other  alternative 
than  a  formal  demonstration  test  program  at  a  later  point  in  the  program 
but  prior  to  the  release  to  production  decision  (Milestone  III) . 
e.   Updating  IOT&E  Planning 

The  early  planning  required  to  support  initial  DCPs  and  Mile- 
stone I  will  require  frequent  updating  to  keep  pace  with  a  dynamic  pro- 
gram.  The  participating  field  activity  Technical  Manager  and  contractor 
will  usually  take  the  lead  in  maintaining  the  currency  of  the  DT&E  program 
but  the  Program  Manager  or  T&E  coordinator  may  have  to  provide  the 
stimulus  for  the  update  of  IOT&E  planning  during  the  early  R&D  phases. 
Few  ALGM  system  programs  have  had  adequate  user  participation  with  the 
desirable  continuity  of  personnel  to  minimize  the  problems  of  transition 
from  DT&E  to  OT&E  to  deployment/utilization.   The  early  evaluations  may 
be  largely  restricted  to  mock-up  exercises  and  user-evaluator  participa- 
tion in  the  R&D.   One  of  the  user-evaluator ' s  major  activities  should  be 
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to  use  the  day-to-day  exposure  to  the  R&D  system  as  on-the-job  training 
in  preparation  for  the  IOT&E  program.   Good  IOT&E  planning  should  cover 
the  allocation  of  manpower  spaces  or  billets,  assignment  of  personnel 
and  their  training,  equipment  provisioning  (including  at  least  pre- 
liminary technical  manuals),  ground  support  equipment,  spare  components, 
launchers,  and  missiles.   IOT&E  planning  may  also  include  plans  and/or 
requirements  for  instrumentation,  ground  tests,  and  flight  tests  of 
important  profiles  and  will  also  provide  for  data  collection,  analysis, 
and  reporting.   The  latter  assumes  even  greater  significance  when  the 
Integrated  Test  Program  approach  is  followed  and  the  reliability  data 
base  is  to  be  cumulative  during  crucial  DT&E  and  IOT&E  tests  and  evalua- 
tions . 

f.   Computer  Software 

As  ALGM  designs  have  become  more  sophisticated,  utilization 
of  mini-computers  has  become  more  commonplace  and  the  problem  of  T&E  of 
the  central  processor  unit  and  the  associated  software  has  grown.   The 
Program  Manager  or  T&E  coordinator  must  ensure  that  software  products 
are  tested  appropriately  during  each  phase.   Even  though  the  computer  is 
a  critical  component  of  the  ALGM,  software  has  often  been  developed  more 
as  an  add-on  than  as  an  integral  part  of  the  overall  system.   Software 
requirements  need  the  same  T&E  consideration  as  hardware  requirements 
in  the  early  program  phases.   It  is  more  difficult  to  determine  the 
status  of  completion  of  various  phases  of  the  software  program  (as  com- 
pared to  hardware  programs) ,  so  it  is  important  to  explore  how  contractors 
develop  and  test  software  programs.   No  standard  procedure  seems  to  be 
available  within  DoD  for  orderly  T&E  of  software  items,  yet  the  increased 
percentage  of  ALGM  development  cost  introduced  by  software  makes  the 
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establishment  of  a  suitable  procedure  a  matter  of  utmost  importance. 
Any  new  procedure  should  provide  for  orderly  program  definition  and 
for  continuous  testing  and  monitoring  of  the  software  program  develop- 
ment, to  provide  assurance  that  adequate,  efficient,  reliable  operation 
will  be  possible.   Annex  A  to  the  Defense  Science  Board  Report  provides 
specific  guidelines  for  tracking  the  development  of  computer  programs 
essential  to  the  functioning  of  weapon  systems  [Ref .  14] . 
g.   System  Reliability 

Inability  to  meet  specified  reliability  requirements  and 
poor  operational  reliability  has  characterized  many  recently  developed 
ALGMs.   It  should  be  emphasized  that  poor  reliability  is  not  only  a 
function  of  component  failures  but  also  is  influenced  greatly  by 
failures  induced  by  poor  hardware  design,  poor  software  design,  operator 
errors,  wear  out  of  mechanical  components,  and  failure  to  appreciate 
the  severity  of  operational  environmental  conditions.   The  Program 
Manager  or  T&E  coordinator  must  require  that  realistic  operational 
as  contrasted  with  design  (inherent)  reliability  requirements  be  defined, 
in  terms  of  completing  a  mission  of  specified  duration  under  stated 
mission  (environmental)  conditions,  and  that  testing  adequate  to  demon- 
strate achievement  of  these  requirements  be  accomplished  successfully. 

It  is  necessary  that  the  improvement  of  reliability  be  a 
planned  activity  during  the  development  phases  of  the  program,  that  it 
be  monitored  during  these  phases,  and  its  achievement  proven  by  testing 
prior  to  the  major  production  decision.   Interim  goals  must  be  estab- 
lished, with  tests  devised  based  on  these  goals,  to  allow  tracking  of 
reliability  growth  throughout  the  program.   This  progressive  attainment 
of  interim  goals  toward  the  required  reliability  for  the  ALGMmust  be 
reviewed  at  critical  points  or  milestones  of  the  program,  as  follows: 
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(1)  At  the  time  the  Program  Manager  requests  initiation  of 
engineering  or  full-scale  development  (Milestone  II)  ,  he 
should  be  prepared  to  show  a  realistic  reliability 
growth  plan  with  sufficient  test  time  and  resources  to 
achieve  the  program  reliability  requirements  from  initial 
and  interim  goals  to  final  requirement. 

(2)  At  the  time  the  Program  Manager  requests  initiation  of 
limited  production  (Milestone  II  A) ,  he  should  be 
prepared  to  show: 

(a)  By  contractor  demonstration  test  results,  the  sys- 
tem has  achieved,  at  a  reasonable  confidence  level, 
the  expected  percent  of  the  reliability  requirement 
for  the  ALGM,  where  both  confidence  level  and  per- 
cent achievement  are  appropriate  to  the  program 

at  this  stage,  i.e.,  tracking  projected  reliability 
growth  rate. 

(b)  There  still  remains  in  the  development  program 
sufficient  ALGM  system  testing  time  to  carry  on 
reliability  growth  from  the  point  achieved  to  the 
program  reliability  requirement. 

(3)  At  the  time  the  Program  Manager  requests  authorization 
for  full-scale  production  (Milestone  III) ,  he  should  be 
prepared  to  show: 

(a)   By  contractor  demonstration  and  accumulated 

TECHEVAL  and  IOT&E  test  results,  the  ALGM  system 
has  achieved,  at  a  reasonable  confidence  level, 
the  program  minimum  acceptable  reliability  require- 
ment. 
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(b)  A  management  plan,  test  plan,  and  funds  to  utilize 
the  remaining  test  time  in  the  development  program 
for  a  vigorous  program  of  reliability  growth. 

(c)  A  plan  for  some  of  the  earliest  production  missiles 
to  be  allocated  to  early  fleet  test  programs  for 
reliability  and  life  tests.   These  ALGMs  should  be 
required  to  accumulate  many  more  ramp  exposure 
hours  and  captive  flight  hours  than  usual  prior  to 
launch.   In  this  fashion,  early  fleet  test  missiles 
can  provide  confirmation  of  ALGM  reliability  or 
provide  early  indications  of  production  problem 
areas  and  possibly  preclude  dangerous  situations 

or  stand-downs  of  important  capabilities. 
h.   Environmental  Determination  Effort 

To  confirm  the  operational  environmental  requirements  con- 
tained in  ALGM  specifications  and  the  defined  mission  profile  and  to 
support  the  development  laboratory  tests  and  operational  tests,  thorough 
measurements  should  be  made,  as  early  as  hardware  permits,  to  determine 
the  actual  environment  in  which  the  ALGM  components  must  live  during  the 
transportation,  captive  flight,  launch  and  free  flight  phases  of  the 
ALGM  life  cycle.   These  environmental  measurements  should  be  conducted 
during  advanced  development  preferably  and  no  later  than  early  engineer- 
ing development,  because  of  the  potential  impact  on  the  basic  ALGM  design 
and  on  the  test  programs.   Funds,  hardware,  and  necessary  time  should  be 
provided  for  the  measurement  effort.   In-flight  environments,  especially 
vibration,  temperature,  shock  and  stress  imposed  during  the  operational 
use  of  the  ALGM,  should  be  measured  and  documented  under  operational 
conditions  using  an  instrumented  ALGM.   The  captive  flight  measurements 
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should  be  utilized  to  structure  ground  environmental  and  reliability 
tests  for  demonstration  of  compatibility  with  captive  flight  require- 
ments under  controlled  laboratory  conditions.   This  has  been  done 
successfully  on  several  recent  ALGM  programs  (e.g.,  D-2  version  of 
Standard  ARM,  AIM-9L  Sidewinder,  and  the  Air  Force's  Maverick) . 
i.   Test  Plan  Coverage 

It  is  important  that  test  plans  prepared  by  the  different 
T&E  participants  support  the  objectives  of  the  Integrated  Test  Program 
by  assuring  compatibility  and  credibility  of  data  bases  so  that  results 
are  integrable  or  capable  of  being  pooled,  configuration  identification 
and  control  so  that  it  is  known  what  was  tested,  and  by  assuring  testing 
is  not  repetitive  or  redundant  and  that  the  "right"  people  are  doing 
the  testing.   The  Program  Manager  or  the  responsible  reviewing  activity 
acting  for  the  Program  Manager,  should  further  assure  that  sub-system 
and  ALGM  system  test  plans  are  compatible  with  pre-established  mile- 
stones and  goals  for  ready  assessment  of  program  progress  at  a  later 
date.   Every  test  plan  should  reflect  the  approved  TEMP  where  appropriate 
and  include  as  a  minimum,  clear  statements  regarding: 

(1)  The  overall  purpose  of  the  test 

(2)  Critical  issues  with  respect  to  operational  requirements 

(3)  The  major  test  objectives 

(4)  The  schedule  of  test  milestone  events 

(5)  The  major  resources  required 

Ca)   Test  environments,  facilities,  and  instrumentation 
(b)   Operational  environment 

(6)  The  organizations  which  will  conduct  the  test  program 

(7)  The  analysis  and  evaluation  approach 
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(8)  Data  acquisition  and  analysis  responsibilities 

(9)  Reporting  of  failures,  failure  analysis  responsibili- 
ties and  procedures  regarding  resumption  of  testing 

Planners  should  ensure  that  the  schedule  will  accommodate  problems  and 
that  clear,  well-defined  milestones  for  review  and  commitment  to  the 
next  phase  have  been  defined. 

The  implementation  of  the  Integrated  Test  Program  approach 
changes  the  traditional  role  of  the  TECHEVAL  agency  by  minimizing 
additional  environmental  testing  as  a  result  of  previously  performed 
contractor  demonstration  tests  and  limiting  the  reliability  testing  to 
that  supporting  the  ALGM  technical  performance  evaluation  (i.e.,  ground 
testing  on  the  aircraft,  captive  flight  tests,  launch  and  free  flight) 
and  contributing  to  the  overall  reliability  data  base.   To  further 
assure   compatibility  with  the  Integrated  Test  Program  approach  and  the 
consistency  and  adequacy  of  the  data  base,  the  TECHEVAL  test  agency 
should  be  required  to  prepare  and  submit  to  the  Program  Manager,  for 
approval,  formal  test  plans  and  procedures  followed  by  a  formal  test 
report  at  the  conclusion  of  TECHEVAL.   OPTEVFOR  personnel  should  support 
the  Program  Manager  and  his  participating  field  activity  Technical 
Manager  in  reviewing  the  test  plans,  procedures,  and  report  and  should 
actively  participate  in  monitoring  the  TECHEVAL  itself  to  assure  com- 
patibility with  operational  requirements  and  consistency  of  data  base. 
2.   Types  and  Categories  of  Tests  and  Resource  Requirements 
a.   Types  and  Categories 

The  types  and  categories  of  tests  relevant  to  ALGM  T&E  are 
discussed  in  section  III.C  and  basically  reflect  the  DT&E/OT&E  structure 
of  OPNAVINST  3960.10.   Figures  8  and  9  depict  DT&E  and  OT&E, 
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respectively,  for  the  Integrated  Test  Program  approach.   Sub-categories 
of  tests  are  identified  and  the  suggested  phase  for  performance  of  the 
test  is  indicated.   This  approach  as  depicted,  is  in  consonance  with  the 
Defense  Science  Board  recommended  hardware  building  block  approach  and 
limits  redundant  testing  between  the  contractor,  TECHEVAL  agency,  and 
OPTEVFOR.   The  common  thread,  in  terms  of  testing,  between  the  partici- 
pants is  obtained  through  tests  performed  on  board  the  aircraft,  on  the 
flight  line,  during  captive  flight,  and  the  launch  and  free  flight  to 
the  target  to  determine  that  the  test  vehicle  meets  its  basic  technical 
requirements.   The  strategy,  when  implementing  the  Integrated  Test 
Program  approach,  is  to  optimize  the  acquisition  of  not  only  performance 
data  but  also  environmental,  reliability,  maintainability,  operability, 
and  human  factors  data  by  appropriate  planning,  design  of  the  tests, 
and  participation  by  user-evaluators  as  well  as  the  developing  agency. 
Under  conditions,  with  appropriate  configuration  management  in  effect 
and  where  no  major  re-designs  after  prototype  development  ALGMs  are 
fabricated,  the  contractor  performance  tests  conducted  under  operational 
conditions  should  yield  data  that  is  integrable  with  TECHEVAL  and  OPEVAL 
data.   The  data,  acquired  under  these  conditions,  is  that  also  relevant 
to  the  ALGM  acquisition  decision  process  and  most  desired  by  DD(TSE)  for 
its  review  and  presentation  to  the  DSARC,  because  it  reflects  progres- 
sive attainment  of  reliability  goals  and,  ultimately,  specified  require- 
ments under  operational  conditions  [Ref .  14] .   It  also  provides  infor- 
mation relative  to  ALGM  maintainability  and  operability  and  can  also 
yield  much  data  useful  for  human  factors  analysis  (e.g.,  poor  handling 
characteristics,  poor  placement  of  controls  and  readouts). 
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b.   ALGM  Models 

Figure  10  depicts  the  different  configurations  of  ALGMs  that 
will  be  required  during  the  different  phases  of  the  development  program. 
The  tactical  ALGMs  (live  warhead)  will  be  primarily  used  during  OPEVAL 
and  will  also  be  utilized  to  a  limited  extent  during  TECHEVAL  and  contrac- 
tor testing.   This  limitation  is  due  primarily  to  the  need  for  acquiring 
telemetered  performance  and  environmental  data  with  an  instrumented  test 
section  throughout  the  DT&E  period  but  also  is  due  to  the  difficulty  and 
expense  of  conducting  warhead  shots  and  the  paucity  of  target  assets. 
Guided  Test  Vehicles  (GTVs)  are  used  to  evaluate  full  seeker  and  control 
section  capabilities  in  captive  and  free  flight  with  the  aid  of  the 
instrumented  test  section.   Particular  emphasis  is  directed  toward  seeker/ 
control  interface  and  launch  and  guidance  characteristics  to  the  target. 
Flight  Test  Vehicles  (FTVs)  are  used  to  evaluate  aerodynamic  features 
and  flight  control  and  autopilot  designs,  and  usually  do  not  include  a 
complete  or  functional  ALGM  seeker  section.   Static  Test  Vehicles  (STVs) , 
featuring  an  inert  warhead  and  rocket  motor  but  with  full  capability 
seekers  and  control  sections,  are  intended  for  extended  periods  of 
laboratory  investigation  and  testing,  i.e.,  environmental  evaluations 
and  tests  and  reliability  growth  testing.   The  Environmental  Determina- 
tion Vehicle  (EDV)  is  specially  instrumented  to  acquire  temperature, 
vibration,  shock,  and  acoustic  data  during  captive  flight  on  representa- 
tive user  aircraft.   The  same  EDV  will  be  used  in  the  laboratory  to 
simulate  in-flight  conditions  and  to  develop  environmental  and  relia- 
bility test  criteria  and  should  be  available  throughout  the  development 
program  in  the  event  of  application  to  other  user  aircraft  or  major 
internal  ALGM  changes  occur.   Dummy  ALGMs  are  used  primarily  for  handl- 
ing tests,  container  tests,  evaluating  interfaces,  and  static  displays. 
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c.   Suggested  Test  Sequences/Flows 

Figures  11,  12,  and  13  provide  suggested  test  sequences/flows 
for  the  contractor  tests,  TECHEVAL,  and  OPTEVFOR,  respectively,  to  maxi- 
mize the  impact  of  the  Integrated  Test  Program  approach.   A  sequence/flow 
is  not  shown  for  DT-I  because  during  this  experimental  period  test  se- 
quence/flow is  normally  unstructured,  has  very  few  test  assets,  and 
usually  features  the  use  of  hybrid  ALGMs.   For  example,  a  recent  Anti- 
Radiation  Missile  (ARM)  program  used  an  experimental  seeker  and  parts  of 
Sparrow  and  SHRIKE  missiles  along  with  a  SHRIKE  rocket  motor  for  its 
early  captive  flight  and  FTV  investigations. 

Also  shown  within  the  individual  activity  blocks  are  typical 
ALGM  asset  quantities  for  that  particular  group  of  activities.   Such 
asset  quantities  should  normally  provide  adequate  numbers  for  required 
tests  yet  retain  a  modicum  of  flexibility  in  the  event  of  test  failure 
or  other  setback.   For  example,  14  advanced  development  ALGMs  are  shown 
distributed  as  follows:   10  for  captive  flight,  launch  and  free  flight 
testing,  2  for  environmental  survey  tests  followed  by  failure  mode  test- 
ing, and  2  for  environmental  determination  flights  and  laboratory  con- 
firmation testing. 

The  important  features  to  be  observed  from  the  Integrated 
Test  Program  approach  are: 

(1)  Progressive  attainment  of  goals  and  requirements  by 
the  appropriate  agency  utilizing  optimum  assets. 

(2)  As  shown  in  Figures  11,  12,  and  13,  TECHEVAL  and 
OPTEVFOR  personnel  participate  in  the  contractor  test 
effort,  and  OPTEVFOR  participates  in  the  TECHEVAL  test 
effort  to  provide  the  proper  operational  user-evaluator 
influence  on  test  structuring  and  performance. 
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(3)  The  contractor  does  what  he  has  been  contractually  obli- 
gated to  do  with  performance  of  development  tests  and 
Contractor  Demonstration  Tests  (CDTs) . 

(4)  A  controlled,  consistently  planned,  executed  and  acquired 
data  base  is  established  with  the  CDTs  and  built  upon 

by  TECHEVAL  and  OPEVAL  with  an  increasing  operational 
flavor  through  TECHEVAL  and  OPEVAL  (where  only  tactical 
ALGMs  are  used) . 

(5)  The  TECHEVAL  role  is  modified  and  limited  to  confirming 
ALGM  specification  compliance  with  ground  tests  on  the 
aircraft  and  captive  flights,  launches  and  free  flights 
and  also  performs  a  supportability  evaluation  (no 
duplication  of  already  accepted  laboratory  environmental 
tests  or  reliability  tests) . 

(6)  Failure  reporting,  analysis,  and  corrective  action 
determination  and  verification  criteria  should  be 
uniformly  applied  and  implemented  throughout  the 
development  program  by  the  Program  Management  staff. 

d.   Resource  Requirements 

ALGMs  cover  a  broad  spectrum  from  Short  Range  Air- to- Air 
(SRAAM)  to  High  Speed  Anti-Radiation  Missiles  (HARM)  with  air-to-air  and 
air-to-ground  capabilities.   Many  different  kinds  of  facilities,  chambers 
and  targets  or  target  simulators  are  required  to  support  these  ALGM  T&E 
efforts.   Because  of  this  wide  dispersion  of  requirements,  the  resource 
requirement  will  first  be  presented  in  a  general  discussion  framework 
and  then  requirements  for  a  typical  ALGM  program  will  be  shown.   Generally, 
resource  requirements  for  ALGMs  exclusive  of  T&E  personnel  requirements 
encompass  the  following: 
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(1)  Contractor  facilities  and  chambers  for  sub-systems  and 
ALGM  system  level  performance,  structural,  and  a  full 
range  of  environmental  tests.   Figure  14  is  typical  of 
the  environments  to  which  ALGMs  must  be  subjected. 

(2)  Contractor  GSE/PGSE  to  support  the  testing  along  with 
necessary  tools,  benches,  stands,  and  handling  equipment, 

(3)  Government  facilities  for  assembly,  checkout,  and  inte- 
gration testing  of  ALGMs  prior  to  captive  flight  test- 
ing and  launch. 

(4)  Ranges  and  range  instrumentation  compatible  with  the 
ALGM  and  its  mission  and  possessing  sufficient  flexi- 
bility to  permit  exploration  of  the  performance  enve- 
lopes in  an  operational  environment. 

(5)  Test  aircraft  including  using  or  ALGM  carrying  aircraft 
representative  of  that  with  which  the  ALGM  will  be 
deployed,  chase  aircraft  for  safety  and  photographic 
purposes,  as  well  as  drones  for  use  until  the  ALGM 
rocket  motor  is  man-rated. 

(6)  Instrumentation  pod  or  an  on-board  installation  to 
support  the  captive  flight  environmental  determination 
effort. 

(7)  Target  simulators,  aerial  targets,  and  target  compexes 
for  air-to-ground  ALGMs. 

The  foregoing  is  an  abbreviated,  but  representative,  indication  of  re- 
source requirements  for  ALGM  T&E.   Tables  II  and  III  reflect  the 
resource  and  facility  requirements  for  a  typical  ALGM  development  T&E 
program  which  are  beyond  those  normally  available  for  fabrication, 
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Test  Item 

Environmental 

Test 

Seeker 

Control 

ALGM1 

T 

D 

W&F 

Radome 

0 

N 

0 

N 

0 

N 

0 

N 

Acceleration 

Captive  Carry 

X 

X 

X 

X 

Ignition 

X 

X 

X 

Acoustical  Noise 

X 

X 

X 

Aero  Heating 

Captive  Carry 

X 

X 

X 

Free  Flight 

X 

X 

X 

Altitude 

X 

X 

X 

Fust 

X 

X 

EMI 

X 

Humidity 

X 

X 

X 

Immersion 

^X 

Rain 

X 

Rain  Erosion 

X 

Salt  Fog 

X 

X 

X 

Shock 

Captive  Carry 

X 

X 

X 

X 

Ignition 

X 

X 

X 

X 

Temp.  High 

x 

X 

X 

X 

Temp. ,  Low 

X 

X 

X 

X 

Temp.  Shock 

X 

X 

X 

X 

Vibration 

Captive  Carry 

X 

X 

X 

X 

Free  High* 

X 

X 

X 

X 

Vibration,  Trans. 

X 

X 

X 

X 

Hot  C-as  Tests 

X 

X 

Structural  load. 

^x 

^x 

X 

Temp. /Altitude 

3x 

NOTES 

C  =  Cperating,  during  part  of  the  test,  the  test  item  may  be 

nonoperating. 
N  =  Nonoperating. 

1  =  Includes  Seeker,  Warhead,  Control,  Inert  Rocket  Motor, 

unless  otherwise  specified. 

2  =  Test  to  be  conducted  with  dummy  missile(s). 

3  =  Conducted  on  short  round  (Seeker,  Control,  Inert  Warhead 

Sections  only). 


Figure  14.  ALGM  Environmental  Test  Matrix 
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TABLE  II.   Government  Test  Facilities  Requirements 
(Adapted  from  Ref.  21) 


Base/Center 

Naval  Weapons  Center 
China  Lake,  California 


Facility 

Missile  Integration 
Facility  (Bldg  31420) 

Til  Ground  Station  (Shrike 
Assembly  Bldg) 

Hangar  3  (NAF) 


MARTS  Facility (Bldg. 31420) 

A-7C  Project  A/C  Data 
Reduction  Facility 
(Bldg.  31420) 

Missile  Assembly  Bldg. 


Purpose 

Missile  Assembly 
Test 


Aircraft  Integration 
Test 

Seeker/Avionics  Tests 

A-7C  Data  Reduction 


Final  Missile  Inte- 


(Area  R  Test  Facility  Area)  gration  and  Test 


Echo  Range 
George  Range 
NODAC  Facility 
Range  Safety 

Propulsion  Firing  Bay 
Warhead  Ranges 

A-7E  Integration  Labora- 
tory (Hangar  3) 

SNORT  Facility 

Beacon  Shop 

T-Range 

Anechoic  Chamber 
(Bldg.  31420) 

Environmental  Test  Facili- 
ties (Michelson  Laboratory) 

Analog  Simulation 
Laboratory 


Captive  Flight  Test 

Captive/Firing  Tests 

Data  Reduction 

Command  Destruct 
System  Checkout 

Motor  Static  Firings 

Missile/Avionics 
Integration 

Missile/Avionics 
Integration 

Radome  Rain  Erosion 
Tests 

Transponder  Tests 

Hot  Gas  Tests 

Seeker  Tests 

Environmental  Testing 

Control  Section  Tests 


119 


TABLE  II.   Government  Test  Facilities  Requirements  (cont'd) 


Base/Center 

Edwards  Air  Force  Base 
California 


Naval  Air  Test  Center 
Patuxent  River,  Maryland 

Naval  Weapons  Laboratory 
Dahlgren,  Virginia 

Naval  Ordnance  Station 
Indian  Head,  Maryland 


Naval  Weapons  Station 
Yorktown,  Virginia 

Naval  Ammunition  Depot 
Crain,  Indiana 


Facility 
Hangar  Space 

Lab  Space 

APR-38  Integ  Laboratory 

TM  Station 

Launcher  Jettison  Facility 
Carrier  Qualification 
Facility 

Hero  Laboratory 
Warhead  Range 

Explosive  Ordnance  Dis- 
posal (EOD)  Facility 
Propulsion  Firing  Bays 
Environmental  Test 
Facility 

Warhead  Loading  Facility 

Environmental  Facilities 
Test  Arenas 


Purpose 

Missile/Aircraft 
Integ 

Missile  Assy  Test 

Missile/APR-38  Integ 

Missile  Tests 

Jettison  Tests 
Carrier  Launch  and 
Recovery 

Hero  Tests 

Warhead  Design  Proof  Tests 

EOD  Techniques 

Motor  Qualification 

Environmental 

Tests 

Warhead  Loading 
Techniques 

Fuze  Qualification 


TABLE  III.   Government  Test  Aircraft  and  Target 
Requirements  (adapted  from  Ref.  21) 


Platform 

TA-4J  Aircraft  (3/N  152848) 

A-7C  Aircraft  (B/N  156739) 

Chase  Aircraft 

QF-86  Drone 


Purpose 

Missile  captive  and  firing  tests 

Weapon  System  captive  and  firing  tests 

Missile  firing  tests,  photography 

Missile  firing  tests,  FTV-103  (Non- 
man-rated  rocket  motor) 


F-4G/APR-38  Aircraft 

A-7E  Aircraft  (Two  Required) 

NWC  Simulator  Assets  for  Missile  Firings 


Missile  captive  and  firing  tests 

Weapon  System  captive  and  firing  tests 


Expendable  targets  available  at  NWCs  George  Range  to  support  ALGM 
firings  include  simulations  of: 

Targets  and  Target  Complexes 
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assembly,  and  test  at  any  qualified  ALGM  contractor's  plant.   Additional 
information  with  regard  to  T&E  resources,  facilities,  and  funding  of 
these  activities  are  contained  in  section  III  F  and  G  and  Appendix  C 
of  this  thesis. 

3.  Responsibility  for  and  Conduct  of  Tests 

Sections  III  D  and  E  and  V  C  of  this  thesis  address  the  subjects 
of  responsibility  for  tests  and  conduct  of  tests,  respectively,  and  pro- 
vide general  information  with  regard  to  the  roles  of  the  various  T&E 
participants  in  ALGM  development  programs.   The  purpose  of  this  sub-sec- 
tion is  to  extend  the  general  information  mentioned  previously  to  the 
implementation  of  the  proposed  planning  strategy  for  an  Integrated  Test 
Program  and  to  suggest  the  assignment  of  responsibility  to  test  activi- 
ties and  facilities  to  conduct  the  tests  for  ALGM  and  major  component/ 
sections.   Figures  15  and  16  depict  DTSE  and  OT&E  tests  respectively, 
and  the  test  category  numbers  shown  reflect  test  categories  from  Figures 
8  and  9. 

4.  Information  Flow,  Assessment,  and  Reporting 

Section  III  J  of  this  thesis  indicates  the  relative  importance 
of  Information  Flow,  and  Review  to  Navy  T&E  programs 

structured  for  ALGMs .   The  Integrated  Test  Program  approach  of  section 
V  C  and  the  Proposed  Test  Program  Strategy  of  section  VI  do  not  change 
the  historical  reporting  requirements,  responsibilities,  or  channels  of 
information  discussed  in  those  earlier  sections.   It  does  change  the 
traditional  ALGM  T&E  approach  (e.g.,  re-orientation  of  user-evaluator 
participation  and  deletion  of  some  TECHEVAL  laboratory  testing) .   Thus, 
the  Program  Manager  and  COMOPTEVFOR  will  still  be  the  focal  points  for 
reporting  to  higher  authority  within  the  Navy  and  OSD  regarding  DT&E 
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and  OT&E,  respectively.   OSD  will  provide  program  reports  with  T&E 
information  to  Congress,  as  required.   However,  the  information  result- 
ing from  the  implementation  of  the  suggested  strategy  should  give  in- 
creased emphasis  to  the  operational  environment  while  still  providing 
evidence  of  specification  compliance. 

Employment  of  the  suggested  strategy  (sections  VI-A  1,  VI-A  2, 
and  VI-A  3)  should  help  ensure  that  appropriate  information  is  available 
for  reporting  program  status  to  the  DSARC  at  Milestone  II  (release  to 
full-scale  development),  Milestone  II  A  (release  to  pilot  production)  , 
and  Milestone  III  (release  to  full-scale  production) .   The  Program 
Manager  can  point  to  the  planned  program  and  show  progressive  achieve- 
ments relative  to  the  plan  and  OPTEVFOR's  IOT&E  evaluations,  prior  to 
OPEVAL,  should  have  greater  meaning  due  to  the  increased  emphasis  on 
operationally  oriented  T&E.   The  pooled  or  integrated  data  base  will 
provide  the  means  to  greater  visibility  for  all  program  participants 
and  reviewers,  of  problems  that  have  occurred,  steps  taken  to  rectify 
the  problems,  and  the  effectiveness  of  the  corrective  action.   This 
increased  user-evaluator  participation  throughout  the  T&E  program,  the 
plan  for  and  adherence  to  the  progressive  attainment  of  ALGM  perform- 
ance and  reliability  goals/requirements,  and  the  increased  problem 
identification  and  corrective  action  visibility  should  enhance  the 
ALGM  T&E  program  credibility  in  the  eyes  of  DD(T&E)  and  other  reviewers. 
The  resulting  T&E  information  should  be  especially  important  and  use- 
ful to  DD(T&E)  when  evaluating  and  making  his  assessment  of  the  ALGM 
T&E  program  to  the  DSARC. 
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B.   ALGM  T&E  ASSESSMENT 

1.   Impact  of  New  Requirements  and  Guidelines 

Several  T&E  efforts  associated  with  recently  developed  ALGMs 
were  examined  to  assess  the  impact  of  the  new  requirements  and  guidelines. 
Unfortunately,  most  of  the  programs  were  already  under  development  by 
contractors  when  the  new  requirements  and  guidelines  were  promulgated, 
and,  as  a  result,  many  deficiencies  which  might  otherwise  have  been 
avoided  have  occurred,  necessitating  redesign,  rework,  and  retest  efforts, 
All  of  the  foregoing  obviously  add  to  the  ALGM  development  cost  and 
lengthen  the  schedule. 

a.   AIM-7F 

For  example,  the  AIM-7F  failed  OPEVAL  I  in  1972  which  re- 
sulted in  a  major  redesign  encompassing  60%  of  the  existing  modules. 
Subsequent  to  the  redesign,  the  TSE  effort  was  not  significantly  re- 
structured from  its  predecessor  (i.e.,  still  performed  to  old  T&E  re- 
quirements) ,  and  a  building  block  approach  to  missile  qualification  was 
not  followed.   The  mission  profiles  had  been  found  to  be  inadequately 
defined  and  the  specifications  and  required  testing  did  not  reflect  the 
operational  environment  in  which  the  AIM-7F  was  to  perform.   Inadequate 
planning  for  T&E  and  poor  coordination  among  T&E  participants  resulted 
in  severe  laboratory  environmental  test  failures  and  failure  attributed 
to  inappropriate  environmental  conditioning  caused  ALGM  flight  failures. 

OPEVAL  II  was  held  in  1974  and  AIM-7F  did  not  pass  because 
captive  flight  MT3F,  while  improved,  was  still  not  adequate  and  design 
specifications  for  successful  guidance  were  not  achieved.   In  fact,  many 
of  the  same  problems  which  occurred  in  OPEVAL  I  were  repeated  in  OPEVAL 
II  indicating  the  intervening  redesign  and  T&E  effort  preceding  OPEVAL 
was  ineffective.   If  the  contractor  DT&E  and  TECHEVAL  effort  had  been 
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properly  planned,  coordinated,  and  implemented,  most  of  the  defi- 
ciencies would  probably  have  been  corrected  prior  to  OPEVAL. 

A  follow-on  version  of  the  AIM-7F  is  now  under  development. 
As  a  result  of  the  preceding  deficiencies,  the  Congressional  Armed  Ser- 
vices Committees  have  taken  a  very  specific  and  direct  interest  in  this 
ALGM's  T&E,  inserting  the  following  language  in  the  FY  1977  appropria- 
tions bill, 

"The  Navy  and  Air  Force  are  advised  to  insure  a  viable  test  pro- 
gram for  the  monopulse  missile  that  will  clearly  demonstrate  the 
ability  of  this  missile  to  perform  in  an  operational  combat  environ- 
ment.  The  Director,  Test  and  Evaluation,  is  to  provide  a  report  to 
the  Committee  on  Armed  Services  at  the  conclusion  of  the  advanced 
development  phase  that  describes  the  test  plan,  the  environment 
(electronic  countermeasures,  etc.),  the  test  conditions,  and  the 
test  results  and  evaluations"  [Ref .  22] . 

b.   AGM-78D 

The  AGM-78D  Standard  ARM  ALGM  presents  a  somewhat  better  pic- 
ture than  the  preceding  due  to  the  implementation  of  an  improved  T&E  pro- 
gram embodying  many  of  the  new  T&E  requirements.   Prior  to  the  "D"  ver- 
sion, Standard  ARM  missiles  had  been  subjected  to  a  limited,  poorly 
planned  and  executed  T&E  effort.   ALGM  specifications  were  inadequate 
(in  fact  were  based  on  the  shipboard  Standard  Missile)  and  requirements 
were  nor.  stated  in  operational  terms  (i.e.,  did  not  reflect  either  the 
physical  or  tactical  environment).   Qualification  of  components,  which 
was  seldom  performed,  was  to  inappropriate  specifications.   The  DT&E 
effort  provided  poorly  for  design  data  feedback  and  was  characterized 
by  poor  failure  analysis  and  reporting.   Contractor  Demonstration  Tests 
(CDTs)  were  performed  for  information  only,  and  there  was  poor  follow- 
up  and  little  corrective  action  when  failures  occurred.   There  was  little 
or  no  coordination  between  the  T&E  participants;  consequently,  when 
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TECHEVAL  was  performed,  ALGMs  were  overstressed  and  failed  due  to  misin- 
terpretation of  the  missile  specification  and  lack  of  a  detailed  test 
plan  and  procedure.   Captive  flight  evaluations  and  ALGM  launches  ex- 
hibited execution  problems  and  results  were  controversial.   OPEVAL  re- 
sults were  similarly  controversial  and  the  missile  failed  to  meet  accept- 
able standards  resulting  in  a  recommendation  to  the  CNO  that  the  AGM-78B 
and  C  not  be  deployed.   They  were  deployed,  however,  because  of  condi- 
tions in  southeast  Asia. 

The  AGM-78D  program,  with  strong  leadership  from  the  Navy 
Program  Manager  and  his  participating  field  activity  Technical  Manager, 
developed  and  implemented  an  improved  T&E  program  starting  in  1972  along 
with  the  redesign  effort.   A  mission  profile  was  jointly  developed  by 
the  Navy  and  the  contractor  based  on  flights  simulating  tactical  missions 
with  partially  instrumented  ALGMs,  and  the  ALGM  specifications  were 
modified  accordingly.   The  building  block  approach  to  component,  assembly, 
section,  and  ALGM  qualification  was  employed  culminating  with  a  formal 
First  Article  Approval  Demonstration  Program  where  environmental  and 
reliability  demonstration  under  simulated  mission  conditions  occurred. 
The  Navy  TECKEVAL  agency  and  the  Air  Force  Test  and  Evaluation  Command 
to  perform  an  OPEVAL  type  of  test  called  AFEVAL  (under  a  joint  test 
agreement  in  which  OPTEVFOR  did  not  participate)  were  involved  in  the 
DT&E  planning  and  monitored  the  Contractor  Demonstration  Tests. 

The  TECHEVAL  was  planned  and  executed  by  the  TECHEVAL  agency 
without  OPTEVFOR  participation.   Asset  and  funding  limitations  were 
present  but  the  TECHEVAL  agency  performed  some  repetition  of  the  Con- 
tractor Demonstration  Tests  (i.e.,  laboratory  environmental  and  ground 
reliability  tests) .   Resource  problems  related  to  range  instrumentation 
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and  target  availability  were  evident,  but  the  results  of  the  captive 
flight  and  launch  performance  and  reliability  evaluations  confirmed 
compliance  with  the  specifications.   The  results  of  the  captive  flight 
reliability  evaluation  and  the  contractor  simulated  captive  flight 
reliability  demonstration  were  remarkably  similar. 

The  AFEVAL,  planned  and  conducted  by  the  Air  Force  under 
the  joint  test  agreement,  exhibited  results  similar  to  TECHEVAL.   Some 
problems  related  to  ground  handling  and  Peculiar  Ground  Support  Equip- 
ment were  encountered  and  operational  testing  flexibility  was  limited 
by  the  necessity  to  use  Research  and  Development  ranges  and  a  shortage 
of  some  resources,  but  ALGM  operational  effectiveness  and  suitability 
were  demonstrated. 

The  foregoing  is  a  classic  example  of  the  situation  where 
we  never  have  enough  time  and  money  to  conduct  a  T&E  program  properly 
the  first  time  but  always  seem  to  be  able  to  find  the  time  and  money 
to  do  it  a  second,  third,  and  even  a  fourth  time. 
2.   Post  OPNAVINST  3960.10 

Currently,  there  is  an  ALGM  development  program  underway  that 
has  the  opportunity  to  develop  and  implement  a  TSE  program  in  accordance 
with  the  new  directives  and  guidelines  of  section  III.   The  AGM-88  HARM 
(High-speed  Anti-Radiation  Missile)  is  currently  in  advanced  development 
and  has  a  system  integration  contractor  under  NAVAIR  contract.   A  develop- 
ment specification  has  been  prepared  with  performance  and  reliability 
requirements  specified  in  terms  of  mission  conditions.   An  extensive 
T&E  planning  effort  has  taken  place  with  considerable  effort  on  the  part 
of  the  Navy  Program  Manager  and  the  other  T&E  participants  to  develop  a 
TEMP  in  accordance  with  the  requirements  of  OPNAVINST  3960.10. 
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Examination  of  the  HARM  preliminary  TEMP  [Ref .  14]  reveals  that 
the  planned  T&E  program  is  moving  in  the  direction  of  the  Integrated 
Test  Program  proposed  in  this  thesis.   Positive  indications  are  extensive 
participation  of  all  T&E  participants  in  the  planning  of  the  T&E  program 
and  integration  of  IOT&E  into  the  DT&E  effort  by  having  OPTEVFOR  (VX-5) 
as  the  operator  in  a  significant  number  of  captive  flights  and  ALGM 
firings.   OPTEVFOR  has  monitored  all  advanced  development  firings  and 
participated  in  all  design  reviews  and  Integrated  Logistic  Support' 
Planning  conferences  as  has  the  TECHEVAL  activity.   A  mission  profile 
has  been  developed  and  is  being  confirmed  by  an  environmental  determina- 
tion effort  consisting  of  captive  flights  with  instrumented  missiles 
followed  by  laboratory  evaluation.   The  mission  profile  is  being  used 
to  structure  the  simulated  captive  flight  environment  for  the  reliability 
growth  test  effort  beginning  in  the  engineering  development  phase  and 
continuing  through  prototype  development.   A  preliminary  view  of  the 
growth  test  plan  has  been  presented  to  NAVMAT  and  DDR&E  (during  a  manage- 
ment review)  and  was  found  acceptable. 

During  the  prototype  development  phase  of  full-scale  development, 
hardware  and  aircraft  installations  will  be  shared  by  DT&E  and  IOT&E. 
Contractor  Demonstration  Tests  for  specification  compliance  are  planned 
for  this  period  including  laboratory  demonstration  of  captive  flight 
reliability. 

The  TECHEVAL  will  be  conducted  utilizing  pilot  production  ALGMs . 
The  current  TECHEVAL  plan  includes  ground  chamber  performance  and 
reliability  testing  as  well  as  captive  flights  to  confirm  specification 
compliance.   There  will  also  be  an  extensive  supportability  evaluation. 
The  reliability  testing  (ground  chamber  and  captive  flight)  will  be  to 
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demonstrate  the  required  specification  minimum  acceptable  MTBF  at  the 
90%  confidence  level. 

OPEVAL  will  include  joint  Navy /Air  Force  testing  to  examine 
operational  effectiveness  and  suitability.   OPTEVFOR  will  conduct  a 
captive  carry  reliability  evaluation  and  will  pool  OPEVAL  data  with 
previous  DT&E/IOT&E  data  in  this  assessment.   Approximately  1000  hours 
of  ALGM  captive  flight  operation  is  planned  during  the  joint  testing 
effort. 

In  summary,  the  AGM-88  HARM  ALGM  T&E  program  approaches  the 
proposed  Integrated  Test  Program  of  this  thesis.   However,  the  HARM 
program  diverges  from  the  Integrated  Test  Program  by  the  TECHEVAL 
planned  conduct  of  environmental  tests  and  TECHEVAL  will  apparently 
not  pool  its  data  with  contractor  and  other  DT&E/IOT&E  data.   OPEVAL 
test  data  will  be  pooled,  as  previously  mentioned,  but  a  significant 
number  of  ALGM  operational  hours  are  to  be  acquired  committing  signi- 
ficant numbers  of  ALGM  assets,  range  facilities,  aircraft,  and 
necessary  support  personnel  for  an  extended  period  of  time. 


130 


VII.   CONCLUSIONS  AND  RECOMMENDATIONS 

A.   CONCLUSIONS 

Consideration  of  the  Navy's  ALGM  T&E  requirements,  management 
problems,  and  alternative  approaches  to  ALGM  T&E  management  leads 
to  the  following  major  conclusions: 

1.  The  successful  outcome  of  the  T&E  program  is  largely  dependent 
upon  the  managerial  ability  of  the  Program  Manager  to  coordinate 
the  diverse  activities  and  resources  all  of  which  are  not  under 
his  direct  control. 

2.  Excessive  duplication  exists  in  testing  performed  by  major 
participants  in  the  Navy  ALGM  T&E  process. 

3.  Operational  test  and  evaluation  agencies  do  not  participate 
adequately  in  early  T&E  planning. 

4.  Although  still  a  relatively  new  requirement,  preparation  and 
implementation  of  TEMPs  for  some  ALGMs  and  other  air  weapon 
systems  are  lagging  behind  program  development  schedules  and 
OPNAVINST  3960.10  guidelines. 

5.  A  number  of  high  level  panels  and  commissions  have  performed 
studies  of  T&E  over  the  past  decade  and  have  made  similar  rec- 
ommendations for  improvement  of  the  process  and  its  management. 
While  policy  changes  have  been  made,  many  of  the  management, 
resource  and  operational  problems  remain. 

Other  significant  conclusions  are: 

1.   Appropriate  test  design  utilizing  statistical  design  of  exper- 
iment techniques  is  not  being  utilized  by  ALGM  T&E  participants 
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resulting,  in  some  cases,  in  excessive  and  inefficient  testing 
as  well  as  improper  interpretation  and  utilization  of  test  data. 

2.  Lack  of  definition  of  or  delay  in  defining  ALGM  mission  profiles 
leads  to  improper  and  often  inadequate  testing  of  ALGMs. 

3.  Central  management  of  NAVAIR  T&E  activities  needs  to  be  strength- 
ened to  reduce  costly  duplication  of  facilities  and  capabilities 
among  the  Navy's  Major  Range  and  Test  Facility  Base  for  air 
weapons. 

4.  Program  Managers  may  be  at  a  disadvantage  when  attempting  to 
resolve,  with  COMOPTEVFOR,  testing  requirements  reflected  in 

the  TEMP  because  of  their  relative  positions  with  respect  to  CNO. 

5.  OPTEVFOR  is  dependent  upon  the  Program  Manager  for  most  of  its 
IOTSE  funding  and  resources;  therefore,  the  structure  of  the 
OT&E  effort  may  be  adversely  affected. 

6.  OPTEVFOR  needs  to  supplement  its  capabilities  and  resources  to 
include  analytical  and  technical  expertise  in  order  to  meet  its 
increasing  testing  responsibilities.   This  is  especially  true  if 
the  trend  toward  increased  simulation  and  reduced  expenditure  of 
ALGM  assets  continues  as  a  result  of  time  and  funding  constraints. 

7.  Software  is  given  insufficient  emphasis  in  early  program  phases, 
as  no  standard  DoD  procedure  for  T&E  of  software  items  exists. 

8.  Unplanned  reviews  and  requests  for  ALGM  T&E  data  or  additional 
tests  by  those  concerned  with  program  review  and  oversight  affect 
program  schedules  and  impact  discretionally  funds. 
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B.   RECOMMENDATIONS 

1.  T&E  for  new  Navy  ALGMs  (ACAT  I  and  II  Programs)  should  be  based 
on  the  Integrated  Test  Program  approach  as  presented  in  this 
thesis . 

2.  Early  T&E  planning  as  recommended  in  the  "Proposed  Test  Program 
Strategy"  should  be  emphasized  on  programs  such  as  Short  Range 
Air-to-Air  Missile  (SRAAM)  and  Medium  Range  Air-to-Air  Missile 

(MRAAM)  which  are  in  advanced  development. 

3.  The  Navy  should  pursue  the  idea  of  utilizing  the  skills  of 
Navy  Laboratories  to  support  OPTEVFOR.   The  laboratories  can 
provide  the  stability  and  continuity  of  personnel  possessing 
appropriate  technical  expertise. 

4.  NAVAIR  should  take  steps  to  improve  the  timely  communication, 
flowdown,  and  implementation  of  new  T&E  directives  and  guide- 
lines. 
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APPENDIX  A 
Organizations  with  which  Interviews  were  Conducted 

1.  Office  of  the  Secretary  of  Defense,  Washington,  D.  C. 

DDRSE(TSE) 

2.  Office  of  the  Chief  of  Naval  Operations,  Washington,  D.  C. 

OP- 98 3 

3.  Headquarters  Naval  Material  Command,  Washington,  D.  C. 

MAT-08 

4.  Naval  Air  Systems  Command,  Washington,  D.  C. 

AIR-06 
AIR-620 
AIR- 242 

5.  Operational  Test  and  Evaluation  Force 

Headquarters,  Norfolk,  Va. 
Air  Development  Squadron  Five 

6.  Naval  Weapons  Center,  China  Lake,  Ca. 

Code  -  06 
Code  -  07 
Code  -  35 

7.  General  Dynamics  Corporation,  Pomona  Division,  Pomona,  Ca. 

Director  of  Defense  Suppression 
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APPENDIX  B 
Descriptions  of  DT&E,  OT&E  and  PAT&E  [Ref.  4] 

1.  Development  Test  and  Evaluation  (DT&E) 

DT&E  is  primarily  that  test  and  evaluation  planned,  conducted,  and 
monitored  by  the  developing  agency  of  the  DoD  component  to  demonstrate 
that  the  engineering  design  and  development  process  is  complete,  that 
the  design  risks  have  been  minimized,  that  the  system  will  meet  its 
performance  specifications,  and  to  estimate  the  system's  military 
utility.   DT&E  includes  testing  of  components,  assemblies,  sub-systems, 
software,  hardware/software  integration,  and  advanced  development  and 
full-scale  system  level  tests  under  various  environmental  conditions. 
Also  tested  is  compatibility  and  interoperability  with  existing  or 
planned  equipments  and  systems. 

2.  Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  that  test  and  evaluation  conducted  to  esimate  the  system's 
military  utility,  operational  effectiveness  (including  survivability 
and  vulnerability) ,  and  operational  suitability  (including  compatibility, 
availability,  interoperability,  reliability,  maintainability,  human 
factors,  logistic  supportability  and  training  requirements)  as  well  as 
the  need  for  any  modifications.   In  addition,  OT&E  provides  information 
on  organization,  personnel  requirements,  doctrine,  and  tactics.   It  may 
also  provide  data  to  support  or  verify  material  in  operating  instruc- 
tions, publications,  and  handbooks.   OT&E  will  be  conducted  in  as 
realistic  an  operational  environment  as  possible.   OT&E  is  divided  into 
two  major  parts:  Initial  OT&E  (IOT&E) ,  which  is  defined  as  all  OT&E 
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prior  to  the  first  major  production  decision;  and  follow-on  OT&E  (FOT&E) , 
which  is  all  OT&E  after  the  first  major  production  decision. 
3.   Production  Acceptance  Test  and  Evaluation  (PAT&E) 

PAT&E  is  test  and  evaluation  of  production  items  to  demonstrate 
that  the  items  procured  fulfill  the  requirements  and  specifications  of 
the  procuring  contract  or  agreement.   It  is  the  responsibility  of  each 
DoD  component  to  accomplish  the  necessary  PAT&E  throughout  the  produc- 
tion phase  of  the  system  life  cycle. 
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APPENDIX  C 
Facilities  for  T&E  of  Navy  ALGM  [Ref.  5] 

1.  Naval  Ordnance  Missile  Test  Facility,  White  Sands,  New  Mexico 

Its  mission  is  to  support  the  Navy  guided  missile  and  rocket  programs 
including  ground  and  flight  testing  and  to  participate  in  the  operation 
of  the  DoD  integrated  missile  test  range  at  White  Sands,  a  major  national 
range.   This  activity  reports  to  CDR,  NAVSEA. 

2.  Naval  Air  Test  Center,  Patuxent  River,  Maryland 

Its  mission  is  to  coordinate  and  perform  T&E  of  aircraft  weapons 
systems,  their  components  and  related  equipment,  conduct  test  pilot 
training,  and  provide  technical  advice  and  assistance  to  NAVAIR,  con- 
tractors, etc.   This  activity  reports  to  CDR,  NAVAIR. 

3.  Pacific  Missile  Range,  Point  Mugu,  California 

Its  mission  is  to  provide  range  support  for  DoD  and  other  designated 
government  agencies  for  launching,  tracking  and  collecting  data  in 
guided  missile,  satellite  and  space  vehicle  research  development  evalu- 
ation and  training  program  and  actual  operations.   This  activity  is  a 
national  range  and  reports  to  CDR,  NAVAIR. 

4.  Naval  Weapons  Center  Ranges,  China  Lake,  California 

Its  mission  is  to  provide  ground  ballistics,  aircraft,  explosive 
testing  and  supersonic  track  ranges  for  testing  (R&D)  the  performance 
of  fuzes,  bombs,  free-fall  weapons,  rockets,  guided  missiles,  and  other 
ordnance  under  actual  operational  conditions .   This  activity  is  part 
of  a  Navy  in-house  laboratory. 
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5.  Naval  Missile  Center,  Point  Mugu,  California 

Its  mission  is  to  perform  test,  evaluation,  development  support  and 
exercise  engineering  cognizance  as  assigned  of  naval  weapons,  weapon 
systems  and  related  devices.   This  activity  reports  to  CDR,  NAVAIR. 

6.  Naval  Weapons  Laboratory  Ranges,  Dahlgren,  Virginia 

Its  mission  deals  primarily  with  testing  of  Navy  guns  and  mounts, 
although  testing  of  some  missile  components  also  occurs.   The  range 
testing  is  subdivided  into  two  broad  categories;  proof  and  acceptance 
testing,  the  object  of  which  is  to  assure  the  quality,  performance, 
safety  and  reliability  of  ordnance  for  the  fleet;  and  developmental  test- 
ing, the  objective  of  which  is  to  provide  an  experimental  basis  for  new 
and  improved  weapons  and  systems.   This  activity  is  part  of  a  Navy  in- 
hous  e  laboratory . 

7.  Naval  Explosive  Ordnance  Disposal  Facility,  Indian  Head,  Maryland 
Its  mission  is  to  conduct  RDT&E  in  technical  matters  for  explosive 

ordnance  disposal  and  render  safe  procedures  for  conventional  and  special 
weapons,  guided  missiles,  biological  and  chemical  munitions,  equipment, 
both  U.S.  and  foreign.   This  activity  reports  to  CDR,  NAVSEA. 
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